ekko's blog

Overview#

Overview — Zephyr Project Documentation

[!Translated from the official website]

Thread Model#

User mode threads are considered untrusted by Zephyr and are therefore isolated from other user mode threads and the kernel. Defective or malicious user mode threads cannot leak or modify private data/resources of another thread or the kernel, nor can they interfere with or control another user mode thread or the kernel.

Examples of Zephyr user mode functionality:

• The kernel can prevent many inadvertent programming errors that could otherwise silently or shockingly compromise the system.
• The kernel can sandbox complex data parsers (such as interpreters, network protocols, and file systems) so that malicious third-party code or data cannot harm the kernel or other threads.
• The kernel can support the concept of multiple logical "applications," each with its own thread group and private data structures, isolating them from each other if one application crashes or is otherwise compromised.

Design Goals#

For threads running in non-privileged CPU state (hereinafter referred to as "user mode"), our goal is to prevent the following:

• We will prevent access to memory that is not specifically authorized, or erroneous access to memory that is incompatible with policy, such as attempting to write to read-only areas.
  • Access to thread stack buffers will be controlled by policy, which partially depends on the underlying memory protection hardware.
    • By default, user threads can read/write access their own stack buffers.
    • User threads can never access the stack of user threads that do not belong to the same memory domain.
    • By default, user threads can never access the thread stack owned by supervisor threads, or the thread stack used for handling system call privilege escalation, interrupts, or CPU exceptions.
    • User threads can read/write access the stacks of other user threads in the same memory domain, depending on the hardware.
      • On MPU systems, threads can only access their own stack buffers.
      • On MMU systems, threads can access any user thread stack in the same memory domain. Portable code should not assume this.
  • By default, all threads within the kernel scope can access program text and read-only data in a read-only manner. This policy can be adjusted.
  • Aside from the above, user threads are by default not allowed to access any memory.
• We prevent the use of unauthorized device drivers or kernel objects based on the permission granularity of each object or each driver instance.
• We will validate kernel or driver API calls with erroneous parameters, which could otherwise lead to kernel crashes or corruption of kernel-specific data structures. This includes:
  • Using the wrong type of kernel object.
  • Using parameters that are out of the appropriate range or unreasonable values.
  • Passing memory buffers that the calling thread does not have sufficient permissions to read or write, according to the API's semantics.
  • Using kernel objects that are not in the correct initialized state.
• We ensure detection and safe handling of user mode stack overflows.
• Prevent system calls that exclude kernel configuration features.
• Prevent disabling or tampering with memory protection defined by the kernel and enforced by hardware.
• Prevent re-entering supervisor mode from user mode, except through kernel-defined system calls and interrupt handlers.
• Prevent user mode threads from introducing new executable code, except in cases supported by kernel system calls.

We specifically do not guard against the following attacks:

• The kernel itself and any threads running in management mode are assumed to be trusted.
• It is assumed that the toolchain and any supplementary programs used by the build system are trusted.
• It is assumed that the kernel build is trusted. There is considerable build-time logic to create a valid kernel object table, define system calls, and configure interrupts. The .elf binaries used in this process are all assumed to be trusted code.
• We cannot prevent errors in memory domain configurations completed in kernel mode, which could expose private kernel data structures to user threads. The RAM of kernel objects should always be configured as supervisor-only.
• User mode threads can be top-level declared and permissions assigned to kernel objects. Generally, all C and header files that are part of the kernel build that generates zephyr.elf are assumed to be trusted.
• We do not prevent DDOS attacks through thread CPU shortages. Zephyr does not have thread priority aging, and user threads of a specific priority can starve all lower-priority threads, and if time slicing is not enabled, can also starve other threads of the same priority.
• There is a build-time defined limit on the number of threads that can be active simultaneously, beyond which creating new user threads will fail.
• Stack overflows of threads running in management mode may be caught, but the integrity of the system cannot be guaranteed.

Policy Details#

In summary, we achieve these thread-level memory protection goals through the following mechanisms:

• Any user thread can only access a subset of memory: typically its stack, program text, read-only data, and any partitions configured in the memory protection design it belongs to. Access to other memory must be done on behalf of the thread through system calls, or specifically granted by the supervisor thread using the memory domain application interface. Newly created threads inherit the memory domain configuration of their parent thread. Threads can communicate by sharing membership in the same memory domain or through kernel objects (such as semaphores and pipes).
• User threads cannot directly access memory belonging to kernel objects. While pointers to kernel objects can be used to reference these objects, actual operations on kernel objects are done through the system call interface. Device drivers and thread stacks are also considered kernel objects. This ensures that any data belonging to the kernel private in kernel objects cannot be tampered with.
• By default, user threads do not have permission to access any kernel objects or drivers other than their own thread objects. This access must be granted by another thread in supervisor mode, or that thread must simultaneously have permission to access both the receiving thread object and the kernel object being granted access. When creating a new thread, it can be optionally set to automatically inherit all kernel object permissions granted by the parent thread, except for the parent thread itself.
• For performance and space considerations, Zephyr typically performs little or no parameter error checking on kernel objects or device driver APIs. Accessing from user mode through system calls involves an additional layer of processing functions that need to strictly validate access permissions and object types, check the validity of other parameters through boundary checks or other methods, and validate correct read/write access to the relevant memory buffers.
• The way thread stacks are defined is such that exceeding the specified stack space will cause a hardware fault. The specifics vary by architecture.

Note#

If all kernel objects, thread stacks, and device driver instances are to be used in user mode, they must be defined at build time. Dynamic use cases for kernel objects require predefined available object pools.

If additional application binary data is loaded after the kernel starts, there will be some restrictions:

• Loaded object code will not be able to define any kernel objects recognizable by the kernel. Instead, this code needs to request kernel objects from the pool using the API.
• Similarly, since the loaded target code is not part of the kernel build process, this code cannot install interrupt handlers, instantiate device drivers, or define system calls, regardless of the mode in which it runs.
• If the loaded target code does not come from verified source code, it should always be entered while the CPU is in user mode.

Routine Description#

ekko@work: ~/zephyrproject/zephyr/samples/userspace dev!
$ tree                                                                                                               
.
├── hello_world_user
│   ├── CMakeLists.txt
│   ├── README.rst
│   ├── prj.conf
│   ├── sample.yaml
│   └── src
│       └── main.c
├── malloc
│   ├── CMakeLists.txt
│   ├── prj.conf
│   └── src
│       └── main.c
├── prod_consumer
│   ├── CMakeLists.txt
│   ├── README.rst
│   ├── prj.conf
│   ├── sample.yaml
│   └── src
│       ├── app_a.c
│       ├── app_a.h
│       ├── app_b.c
│       ├── app_b.h
│       ├── app_shared.c
│       ├── app_shared.h
│       ├── app_syscall.c
│       ├── app_syscall.h
│       ├── main.c
│       ├── sample_driver.h
│       ├── sample_driver_foo.c
│       └── sample_driver_handlers.c
├── shared_mem
│   ├── CMakeLists.txt
│   ├── README.rst
│   ├── boards
│   │   ├── nucleo_f746zg.overlay
│   │   └── ok3568.conf
│   ├── prj.conf
│   ├── sample.yaml
│   └── src
│       ├── enc.c
│       ├── enc.h
│       ├── main.c
│       └── main.h
└── syscall_perf
    ├── CMakeLists.txt
    ├── README.rst
    ├── prj.conf
    ├── sample.yaml
    └── src
        ├── main.c
        ├── test_supervisor.c
        ├── test_user.c
        └── thread_def.h

11 directories, 43 files

There are a total of four official use cases, and the libc test added one use case. The content of each use case:

Development Guide#

This section explains the steps required to use user mode and the operations involved, providing only basic descriptions. For detailed steps, please refer to the prod_consumer example.

1. Project Configuration#

User space needs to be enabled in the Zephyr project file.

[!prj.conf]
CONFIG_USERSPACE=y

2. Definition of Global Variables in User Mode#

User mode has requirements for memory permissions, so global variables used need to be manually allocated to specific partitions:

K_APPMEM_PARTITION_DEFINE(app_a_partition);// Define the partition used by app a

#define APP_A_DATA  K_APP_DMEM(app_a_partition)// Data segment, stores global/static data that is not initialized to 0
#define APP_A_BSS   K_APP_BMEM(app_a_partition)// BSS segment, stores global/static data initialized to 0

// Subsequent definitions of global variables need to prefix with the data segment modifier
APP_A_BSS unsigned int count;
APP_A_DATA uint32_t baudrate = 115200;

3. Create User Thread#

The process is the same as creating a normal thread, with the difference being the option to select K_USER for user mode, and the thread run delay is set to infinite, as there will be other operations on this thread later.

k_tid_t user_tid = k_thread_create(&user_thread, user_stack, STACKSIZE,
			thread_entry, NULL, NULL, NULL,
			-1, K_USER,
			K_FOREVER);

4. Memory Domain Configuration#

Threads in user mode only have permission to operate on partitions within the memory domain. This step also grants the user thread the partition permissions of the previously defined global variables. Multiple partitions can be configured in the memory domain, and it can be dynamically modified. Multiple user threads can achieve shared memory communication and data interaction through a partition that everyone has permission to access.

struct k_mem_domain user_domain;// Define a memory domain

// List of partitions for app a
struct k_mem_partition *app_a_parts[] = {
		&user_hello,
		&z_libc_partition,// libc global variable partition
		&z_malloc_partition// libc malloc partition
	};
// Initialize memory domain
k_mem_domain_init(&user_domain, ARRAY_SIZE(app_a_parts), app_a_parts);

// Add user thread to memory domain
k_mem_domain_add_thread(&user_domain, user_tid);

5. Resource Pool Configuration#

A resource pool is a pointer to a k_heap, which is a heap that can allocate memory. Some API functions require this resource pool for operations. The created thread is not configured with a resource pool by default; privileged mode threads can use the system's own resource pool, while user mode threads need to manually bind a resource pool:

// Global variable defining resource pool
K_HEAP_DEFINE(app_a_resource_pool, 256 * 5 + 128);
// Bind resource pool to thread
k_thread_heap_assign(user_tid, &app_a_resource_pool);

6. Authorizing Kernel Objects#

The definition of kernel objects uses the system heap, which user mode does not have permission to operate on. Therefore, they need to be created during global variable creation (as mentioned in the note at the end of the overview) and then passed to the user thread through authorization:

// Message queue
K_MSGQ_DEFINE(mqueue, SAMPLE_DRIVER_MSG_SIZE, MAX_MSGS, 4);
// Queue
K_QUEUE_DEFINE(queue);
// Driver
APP_A_BSS const struct device *sample_device;
sample_device = device_get_binding(SAMPLE_DRIVER_NAME_0);
// Authorize kernel object list
k_thread_access_grant(user_tid, &mqueue, &queue, sample_device);

7. Run User Thread#

Finally, we reach the last step, after which the thread will run in user mode.

k_thread_start(user_tid);

8. Dynamically Creating Kernel Objects#

If the user thread needs to create some kernel objects for internal use, such as semaphores, queues, locks, etc., rather than obtaining global kernel objects through authorization, the following steps should be followed:

1. Project Configuration#

First, the dynamic object creation feature, which is only available in user mode, needs to be enabled by adding the following option to the project configuration file:

CONFIG_DYNAMIC_OBJECTS=y

2. Ensure Resource Pool is Configured#

This refers to the resource pool configuration mentioned in step 5. The resource pool can be used not only for authorized kernel object API usage but also for dynamically creating kernel objects.

3. Create and Use Kernel Objects#

/*
K_OBJ_MEM_SLAB,
K_OBJ_MSGQ,
K_OBJ_MUTEX,
K_OBJ_PIPE,
K_OBJ_QUEUE,
K_OBJ_POLL_SIGNAL,
K_OBJ_SEM,
K_OBJ_STACK,
K_OBJ_THREAD,
K_OBJ_TIMER,
K_OBJ_THREAD_STACK_ELEMENT, etc.
*/
// Queue
	struct k_queue *queue = k_object_alloc(K_OBJ_QUEUE);
	printf("queue : %p\n",queue);
	k_queue_init(queue);
	printf("k_queue_init finish\n");
	int test = 10;
	
	k_queue_alloc_append(queue,&test);
	printf("k_queue_alloc_append finish\n");

	int *get = k_queue_get(queue,K_NO_WAIT);
	printf("get %p : %d\n",get,*get);

	// Lock
	struct k_mutex *mutex = k_object_alloc(K_OBJ_MUTEX);
	printf("mutex : %p\n",mutex);

	k_mutex_init(mutex);
	printf("k_mutex_init finish\n");

	k_mutex_lock(mutex,K_FOREVER);
	printf("k_mutex_lock finish\n");

	k_mutex_unlock(mutex);
	printf("k_mutex_unlock finish\n");

	// Message queue
	struct k_msgq *msgq = k_object_alloc(K_OBJ_MSGQ);
	printf("msgq : %p\n",msgq);

	k_msgq_alloc_init(msgq,1,5);
	printf("k_msgq_alloc_init finish\n");

	k_msgq_put(msgq,&test,K_NO_WAIT);
	printf("k_msgq_put finish\n");
	int recv = 0;
	k_msgq_get(msgq,&recv,K_NO_WAIT);
	printf("recv : %d\n",recv);
	// Semaphore
	struct k_sem *sem = k_object_alloc(K_OBJ_SEM);
	printf("sem : %p\n",sem);

	k_sem_init(sem,0,10);
	printf("k_sem_init finish\n");

	k_sem_give(sem);
	printf("k_sem_give finish\n");

	k_sem_take(sem,K_NO_WAIT);
	printf("k_sem_take finish\n");

	// Stack
	struct k_stack *stack = k_object_alloc(K_OBJ_STACK);
	printf("stack : %p\n",stack);

	k_stack_alloc_init(stack,10);
	printf("k_sem_take finish\n");

	k_stack_push(stack,1);
	printf("k_sem_take finish\n");

	stack_data_t pop;
	k_stack_pop(stack,&pop,K_NO_WAIT);
	printf("pop : %ld\n",pop);

4. Notes#

Not all APIs can be used normally. The examples shown above can be used normally, but a complete list of APIs that can be used for dynamically created objects in user mode has not been found in the official documentation. Reference materials include Memory Protection Design #thread-resource-pools— Zephyr Project Documentation and Kernel Objects — Zephyr Project Documentation.

Current Usage#

The previous steps have already explained the parts that can currently be used:

1. Usage of libc library
2. Usage of globally authorized variables in partitions
3. Authorized usage of kernel objects
4. Dynamically creating kernel objects

Advantages and Disadvantages Analysis#

Advantages#

The various advantages of user mode features have been explained in the overview section. This part will discuss several possible scenarios in practical use:

1. User thread crashes will not cause the system to crash; a monitoring thread can monitor the operation of each user thread, restart the corresponding thread after a crash, ensuring that the operating system itself is not affected.
2. User thread permissions are limited, and multi-team, multi-user APP development will not affect other parts even if errors occur, providing good sandbox isolation.
3. Memory permissions are clear, which can avoid many programming errors during the development process.

Disadvantages#

1. The process is relatively complex, raising the threshold.
2. Permissions are limited; some POSIX-related interfaces encapsulate many kernel objects. Currently, a good way to authorize has not been found, and only kernel objects defined by Zephyr can be used for authorization, increasing dependency on Zephyr kernel objects.
3. Performance loss; the previously mentioned system call performance test case is for the RISC-V platform. The following output is for reference, listing the differences in clock cycles and instruction counts:
   User thread: 18012 cycles 748 instructions
   Supervisor thread: 7 cycles 4 instructions
   User thread: 20136 cycles 748 instructions
   Supervisor thread: 7 cycles 4 instructions
   User thread: 18014 cycles 748 instructions
   Supervisor thread: 7 cycles 4 instructions

This article is just a preliminary summary after testing user mode. For detailed information, please refer to the official website and example programs. User Mode — Zephyr Project Documentation