It used to be enough to create /etc/X11/Xwrapper.config with the following in it

allowed_users = anybody
needs_root_rights = no

That is no longer enough. So patch X.

Place this following in /etc/portage/patches/x11-base/xorg-server/console.diff
(version 1.20.8 tested but probably works on others ?)
If you’re not using Gentoo then you should be 😉
You’ll need to google how to patch and compile packages on your distro.

diff -Naur a/hw/xfree86/xorg-wrapper.c b/hw/xfree86/xorg-wrapper.c
--- a/hw/xfree86/xorg-wrapper.c	2020-03-29 21:21:15.000000000 +0100
+++ b/hw/xfree86/xorg-wrapper.c	2020-05-22 14:23:59.825069376 +0100
@@ -206,29 +206,6 @@
 
     parse_config(&allowed, &needs_root_rights);
 
-    /* For non root users check if they are allowed to run the X server */
-    if (getuid() != 0) {
-        switch (allowed) {
-        case ROOT_ONLY:
-            /* Already checked above */
-            fprintf(stderr, "%s: Only root is allowed to run the X server\n", argv[0]);
-            exit(1);
-            break;
-        case CONSOLE_ONLY:
-            /* Some of stdin / stdout / stderr maybe redirected to a file */
-            for (i = STDIN_FILENO; i <= STDERR_FILENO; i++) {
-                if (on_console(i))
-                    break;
-            }
-            if (i > STDERR_FILENO) {
-                fprintf(stderr, "%s: Only console users are allowed to run the X server\n", argv[0]);
-                exit(1);
-            }
-            break;
-        case ANYBODY:
-            break;
-        }
-    }
 
 #ifdef WITH_LIBDRM
     /* Detect if we need root rights, except when overriden by the config */