Finally add the sledjchisl C & Lua maanger, and friends.

1 files changed, 503 insertions, 0 deletions

diff --git a/src/sledjchisl/NOTES.txt b/src/sledjchisl/NOTES.txt
new file mode 100644
index 0000000..a5f0861
--- /dev/null
+++ b/src/sledjchisl/NOTES.txt
@@ -0,0 +1,503 @@
+I'm re-purposing this for SledjHamr https://sledjhamr.org/git/docs/index.html
+
+The general structure of SledjHamr is a bunch of servers talking to each
+other via Internet (or just local) connections.  One of them is a web
+server for assets, world data, and inventory. 
+
+Originally I didn't think using a web based world client was a good idea,
+however it might be better to have one, for reasons.  Now I need a web
+management console that can do all the things the current tmux console
+can, including OpenSim console and commands.  Plus account management for
+users.  I can also use a web based Jabber / XMPP front end to chat, IM,
+and group chatter, which would run in the normal viewers web browser. 
+This provides a doorway into putting SledjHamr stuff in existing viewers
+without needing them to support it.  So a web based viewer now makes more
+sense, and also means we can get away with not needing a viewer at all.
+
+Toybox itself doesn't include a web server, and I don't think there is
+one on the roadmap.  So we have to use an external web server, which was
+a design goal of SledjHamr in the first place, using existing mature
+HTTP infrastructure, coz that's already solved problems for a bunch of
+things that plague OS/SL to this day.  Clear your cache!  Pffft.
+
+So sledjchisl.c will be the "love world server", though initially it just
+drives OpenSim_SC in tmux via tmux commands to send keys and read output. 
+Later it might run opensim_SC directly and use STDIN and STDOUT to do
+everything.  It'll also provide the text management front end that runs
+in the left tmux panel of the first window, which is why it's based on
+boxes in the first place.  Later still it can take over opensim_SC
+functions as I move them out of mono.
+
+We will need a text, web, and GUI version of this management front end. 
+Hmmm, maybe don't need a GUI version, GUI users can just run a terminal.
+
+After much research, FastCGI / FCGI seems to be the most portable way of
+interfacing with existing web servers.  FCGI protocol closes STDERR and
+STDOUT, and uses STDIN as it's two way communications channel to the web
+server, so our FCGI module can't be used as the text management front
+end.  This is probably a good idea to keep them seperate anyway, for
+security, coz the web server is exposed to the world, the console isn't.
+
+Currently sledjchisl.c tests to see if it's running in tmux already, if
+it isn't it starts up tmux runs itself into this new tmux, then exits. 
+So it could also test if it's running from FCGI, and switch to web mode,
+then it'll need to find the tmuxed instance to send commands to it. 
+Either via nails connection, or sending tmux commands via shell.
+
+FCGI has methods of dealing with auth and templates.  B-)
+
+So for now I think I'll have the text and web management front ends in
+sledjchisl.c, and the love world server as well.  I can split them up
+later if I need to.
+
+
+
+
+I has   Apache 2.4.25-3+deb9u9
+        MariaDB 10.1.44-MariaDB
+
+
+https://gist.github.com/dermesser/e2f9b66457ae19ebd116
+    Multithreaded example in C.
+
+
+-------------------------------------------------------------------
+
+Apache doesn't seem to support FCGI filter role, so I might have to do
+without.  Might be better anyway.
+
+
+"A Filter is similar in functionality to a Responder that takes a data
+file as a parameter. The difference is that with a Filter, both the data
+file and the Filter itself can be access controlled using the Web
+server's access control mechanisms, while a Responder that takes the name
+of a data file as a parameter must perform its own access control checks
+on the data file."
+
+    Which is fine, our access control checks will be "Is this database
+    defined user already logged on via our FCGI script?".  We should have
+    total control over that.  I was planning on using the FCGI auth
+    mechanism anyway.
+
+
+RESPONDER
+    web server sends FCGI_PARAMS
+        CONTENT_LENGTH
+    web server sends input body FCGI_STDIN
+    fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR
+        it has to finish reading FCGI_PARAMS first
+    fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE)
+
+
+FILTER
+    filtered file has last modified time
+        web server sets FCGI_DATA_LAST_MOD accordingly
+    web server sends FCGI_PARAMS
+        CONTENT_LENGTH   FCGI_DATA_LAST_MOD   FCGI_DATA_LENGTH
+    web server sends input body FCGI_STDIN
+    web servers sends file over FCGI_DATA
+    fcgi app can ignore FCGI_DATA and use it's own cached copy based on FCGI_DATA_LAST_MOD
+    fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR
+        it has to finish reading FCGI_STDIN first, but not FCGI_DATA
+    fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE)
+
+
+Soooo, FILTER might be slower anyway if we are caching the filtered file,
+or mmapping it, coz filter has to start sending the filtered file, even
+if it's to be served from cache.  Plus no need to wait for FCGI_STDIN
+before spewing it out.
+
+
+Last update time for parameters, plus an update frequency.  Once a minute.
+
+    NOTE - SSI is a bit more complex than what I'm currently using.
+        https://en.wikipedia.org/wiki/Server_Side_Includes
+            <!--#include virtual="menu.cgi" -->
+            <!--#include file="footer.html" -->
+            <!--#exec cgi="/cgi-bin/foo.cgi" -->
+            <!--#exec cmd="ls -l" -->
+.           <!--#echo var="REMOTE_ADDR" -->
+            <!--#config timefmt="%y %m %d" -->
+            <!--#config sizefmt="bytes" -->
+            <!--#config errmsg="SSI command failed!" -->
+            <!--#flastmod virtual="index.html" -->
+            <!--#fsize file="script.pl" -->
+            <!--#if expr="${Sec_Nav}" -->
+            <!--#include virtual="secondary_nav.txt" -->
+            <!--#elif expr="${Pri_Nav}" -->
+            <!--#include virtual="primary_nav.txt" -->
+            <!--#else -->
+            <!--#include virtual="article.txt" -->
+            <!--#endif -->
+            <!--#set var="foo" value="bar" -->
+            <!--#printenv -->
+        https://www.w3.org/Jigsaw/Doc/User/SSI.html
+            Adds lots of others, including Java stuff.
+        Mine
+            <!--#lua lua="print(table[key])" -->
+            <!--#lua file="/path/to/script.lua" -->
+            <!--#lua virtual="https://example.com/script.lua" -->
+
+-------------------------------------------------------------------
+
+Account creation process in the database.
+
+Apart from the usual input validation of things...
+
+
+OpenSim/Server/Handlers/UserAccounts/UserAccountServerPostHandler.cs
+    byte[] CreateUser(Dictionary<string, object> request)
+        Looks like their built in web front end, perhaps what is triggered by the console?
+        createdUserAccount
+            = ((UserAccountService)m_UserAccountService).CreateUser(scopeID, principalID, firstName, lastName, password, email, model);
+
+OpenSim/opensim-SC/OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs
+    An XML RPC interface to -
+        private UserAccount CreateUser(UUID scopeID, string firstName, string lastName, string password, string email)
+            account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email);
+            if (userAccountService.StoreUserAccount(account))
+            success = authenticationService.SetPassword(account.PrincipalID, password)
+            gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0));
+            success = inventoryService.CreateUserInventory(account.PrincipalID);
+
+OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs
+    Looks like the console command handler.
+        create user [<first> [<last> [<pass> [<email> [<user id> [<model>]]]]]] - Create a new user
+            protected void HandleCreateUser(string module, string[] cmdparams)
+                Gathers console arguments, or prompts for them.
+                CreateUser(UUID.Zero, principalId, firstName, lastName, password, email, model);
+            public UserAccount CreateUser(UUID scopeID, UUID principalID, string firstName, string lastName, string password, string email, string model = "")
+                Looks almost identical to the OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs one above, but they add -
+                    CreateDefaultAppearanceEntries(account.PrincipalID)
+
+
+
+account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email);
+    OpenSim/opensim-SC/OpenSim/Services/Interfaces/IUserAccountService.cs
+        public UserAccount(UUID scopeID, UUID principalID, string firstName, string lastName, string email)
+            Just holds the data in memory, in a dictionary I think.
+    OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs
+        public bool StoreUserAccount(UserAccount data)
+            Stuffs the data into a new UserAccountData()
+            m_Database.Store(d)
+                As far as I can tell, just dumps this data into the UserAccounts table -
+                    FirstName, LastName, PrincipleID, ScopeID, Email, Created, UserLevel, UserFlags, UserTitle
+                        PrincipleID is their randomly generated with no thought to collisions UUID.
+                        ScopeID is 00000000-0000-0000-0000-000000000000
+                        Userlevel is 0 for most, -1 for Waki, determines if they can log on.  Also higher for gods and things.
+                        UserFlags, I think the only one is "64 god can login to this account using gods password.
+                        UserTitle might default to "Local", or be configurable / and editable.
+                    something something URL encoded "ServiceURLs" mumble
+                        HomeURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GatekeeperURI= InventoryServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f AssetServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f ProfileServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f FriendsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f IMServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GroupsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f
+                        Though most are either NULL, empty, or -
+                            HomeURI= GatekeeperURI= InventoryServerURI= AssetServerURI=
+                    Doesn't metion "active", which is always equal to 1 I guess.
+
+
+
+success = authenticationService.SetPassword(account.PrincipalID, password)
+    OpenSim/Services/AuthenticationService/AuthenticationServiceBase.cs
+        stores password details in "auth" table -
+            UUID
+            passwordSalt = Util.Md5Hash(UUID.Random().ToString());
+            passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt);
+            accountType = "UserAccount";
+            webLoginKey = UUID.Zero.ToString();
+
+
+
+gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0));
+    OpenSim/Services/UserAccountService/GridUserService.cs
+        Stores in database table GridUser
+            HomeRegionID, HomePosition, HomeLookAt
+        The other fields in that table -
+            UserID, LastRegionID, LastPosition, LastLookAt, Online (true or false), Login (timestamp or 0), Logout (timestamp or 0).
+
+
+
+success = inventoryService.CreateUserInventory(account.PrincipalID);
+    OpenSim/Services/InventoryService/XInventoryService.cs
+        Create a bunch of folders in the users inventory, of specific types.
+            rootFolder = ConvertToOpenSim(CreateFolder(principalID, UUID.Zero, (int)FolderType.Root, InventoryFolderBase.ROOT_FOLDER_NAME));
+            XInventoryFolder[] sysFolders = GetSystemFolders(principalID, rootFolder.ID)
+            if (!Array.Exists(sysFolders, delegate(XInventoryFolder f) { if (f.type == (int)FolderType.Animation) return true; return false; }))
+                CreateFolder(principalID, rootFolder.ID, (int)FolderType.Animation, "Animations");
+                    FolderType.BodyPart, "Body Parts"
+                XInventoryFolder folder = CreateFolder(principalID, rootFolder.ID, (int)FolderType.CallingCard, "Calling Cards");
+                folder = CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "Friends")
+                CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "All");
+                    FolderType.Clothing, "Clothing"
+                    FolderType.CurrentOutfit, "Current Outfit"
+                    FolderType.Favorites, "Favorites"
+                    FolderType.Gesture, "Gestures")
+                    FolderType.Landmark, "Landmarks"
+                    FolderType.LostAndFound, "Lost And Found"
+                    FolderType.Notecard, "Notecards"
+                    FolderType.Object, "Objects"
+                    FolderType.Snapshot, "Photo Album"
+                    FolderType.LSLText, "Scripts"
+                    FolderType.Sound, "Sounds"
+                    FolderType.Texture, "Textures"
+                    FolderType.Trash, "Trash"
+
+                    Stores in database inventoryFolders ????
+                        folderName, type, version = 1, folderID = UUID.Random(), agentID = principalID, parentFolderID = parentID
+
+
+
+
+CreateDefaultAppearanceEntries(account.PrincipalID)
+    OpenSim/Services/UserAccountService/UserAccountService.cs
+        protected void CreateDefaultAppearanceEntries(UUID principalID)
+        Creates a bunch of "Default *" body parts and clothes, Ruth 1.0, links them in Inventories current outfit folder.
+        Creates a AvatarWearable[] and puts them all in it.
+        AvatarAppearance ap = new AvatarAppearance();
+            ap.SetWearable(i, wearables[i]);
+        m_AvatarService.SetAppearance(principalID, ap);
+            
+
+
+
+
+UserAccounts table -
+    UserFlags           64 is "allow gods to log in as me"
+                        0xf00 is membershipType, unles there's a title.  Only sent to viewers I think.
+                        32 is Minors   for estate banning purposes.
+                        4 is Anonymous for estate banning purposes.
+                        1 is AllowPublish  in profile, but userprofile has this as separate field.
+                        2 is MaturePublish in profile, but userprofile has this as separate field.
+Presence table -
+    UserID              varchar(255)
+    RegionID            char(36)
+    SessionID           char(36)
+    SecureSessionID     char(36)
+    LastSeen            timestamp
+tokens table (I think this is actually used for something) -
+    UUID                char(36)
+    token               varchar(255)    current example looks like a UUID.
+    validity            datetime
+userdata (empty, can't find any actual usage in the source code, part of profiles) -
+    UserId              char(36)        primary index
+    TagId               varchar(64)     primary index
+    DataKey             varchar(255)
+    DataVal             varchar(255)
+auth.webLoginKey seems to be some sort of passwordy type thing, though perhaps not actually hashed, rarely used, none of IG members have one.
+
+
+PLAN-
+.    username
+.    password
+.    create   login
+
+.check if it's a proper two word name
+.login -> check if it's an existing account, get their UUID.
+    create toke_n_munchie
+    write session record
+
+create -> new user
+    create new UUID
+    check if it's an existing UUID
+        dbCount(, "UserAccounts", "PrincipleID='new-UUID'")
+    loop until we get a new one
+    create toke_n_munchie
+    write session record
+
+
+    Create ->
+    (wait a few seconds before showing this page)
+.    email
+.    email again
+.    password again
+.    DoB
+.    accept terms of service
+.    claim to be an adult
+.    confirm / cancel
+
+    New user
+        UserAccounts.FirstName = ???
+        UserAccounts.LastName = ???
+        UserAccounts.Email = ???
+        UserAccounts.Created = timestamp
+        UserAccounts.PrincipleID = randomly generate UUID, but check for collisions with other accounts.
+                                                            It's a UNIQUE KEY.
+        UserAccounts.ScopeID = 00000000-0000-0000-0000-000000000000
+        UserAccounts.Userlevel = -200
+        UserAccounts.UserFlags = 64
+        UserAccounts.UserTitle = newbie
+        UserAccounts.ServiceURLs = ""
+        UserAccounts.active = 0
+
+        auth.UUID = UserAccounts.PrincipleID
+                                                            It's a PRIMARY KEY.
+        auth.passwordSalt = Util.Md5Hash(UUID.Random().ToString())
+        auth.passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt)
+        auth.accountType = "UserAccount"
+        auth.webLoginKey (varchar(255)) = "00000000-0000-0000-0000-000000000000"
+
+        userdata.UserId = UserAccounts.PrincipleID
+        userdata.TagId = "account creation data"
+                                                            It's a UNIQUE KEY
+        userdata.DataKey = "DoB"
+        userdata.DataVal = ???
+
+        userdata.UserId = UserAccounts.PrincipleID
+        userdata.TagId = "account creation data"
+        userdata.DataKey = "timezone"
+        userdata.DataVal = ???
+
+        userdata.UserId = UserAccounts.PrincipleID
+        userdata.TagId = "account creation data"
+        userdata.DataKey = "Terms of service"
+        userdata.DataVal = "True"
+
+        userdata.UserId = UserAccounts.PrincipleID
+        userdata.TagId = "account creation data"
+        userdata.DataKey = "claims to be an adult"
+        userdata.DataVal = "True"
+
+
+    Validated via email
+    (wait a few seconds before showing this page)
+        UserAccounts.Userlevel = -100
+        UserAccounts.UserTitle = validated
+
+
+    Vouched for
+        userdata.UserId = UserAccounts.PrincipleID
+        userdata.TagId = "vouches"
+        userdata.DataKey = UUID of voucher
+        userdata.DataVal = timestamp of vouching
+
+        UserAccounts.Userlevel = -50
+        UserAccounts.UserTitle = vouched for
+
+
+    Admin approved
+        GridUser.UserID = UserAccounts.PrincipleID
+                                                            It's a PRIMARY KEY.
+        GridUser.HomeRegionID = ???
+        GridUser.HomePosition = ???
+        GridUser.HomeLookAt = ???
+        GridUser.LastRegionID = ???
+        GridUser.LastPosition = ???
+        GridUser.LastLookAt = ???
+        GridUser.Online = False
+        GridUser.Login = 0
+        GridUser.Logout = 0
+
+        UserAccounts.active = 1
+        UserAccounts.Userlevel = 1
+        UserAccounts.UserTitle = Member / Local / whatever
+
+        Load the default IAR.
+
+
+-------------------------------------------------------------------
+
+https://project-awesome.org/aleksandar-todorovic/awesome-c
+    A curated list of C good stuff.
+
+https://wolkykim.github.io/qdecoder/
+    CGI library made by the qlibc guy, does support FCGI.
+    Might be a wrapper around the fcgi_stdio stuff I'm already using?
+
+
+https://danielmiessler.com/study/http/
+    A Security-focused HTTP Primer
+        Nothing much new except to say this about the Referer header -
+            "should not be used to make security decisions as it is controlled by the client"
+            Though others tell us to do precisely that.  lol
+
+
+-------------------------------------------------------------------
+
+apt install libmariadbclient-dev libapache2-mod-fcgid
+
+-------------------------------------------------------------------
+
+Merge it into OpenSim-SC.
+
+    Complication - I had already added the boxes + early sledjchisl.c to opensim-SC.
+        I may have to revert that lot, only a few minor commits, which are already part of the main boxes.
+        d9c772712e27c8e25fab0d17555a9bc11017a125
+        d4ea3e50173df1ad646bdb7dc802f5d320b7e511
+        10ed36b3452ce6373175112716b043047dc896a9
+        2f66c46e7ce18d60cd1f565880fb7762b3399ccb
+        34c5ee4c2a489a506e93d5b303fbc80b263747f0 is the commit that added it.
+        f9bfa831d1ccaa973c42042584510e1a724ddaef the one before it.
+        It's been pushed up to the sledjhamr.org repo.
+
+        git revert d9c772712e27c8e25fab0d17555a9bc11017a125 d4ea3e50173df1ad646bdb7dc802f5d320b7e511 10ed36b3452ce6373175112716b043047dc896a9 2f66c46e7ce18d60cd1f565880fb7762b3399ccb 34c5ee4c2a489a506e93d5b303fbc80b263747f0
+        git remote add -f boxes ~/MyLinux_3/TOYBOX/sledjchisl
+        git merge --allow-unrelated-histories boxes/master
+        git remote rm boxes
+
+
+https://medium.com/altcampus/how-to-merge-two-or-multiple-git-repositories-into-one-9f8a5209913f
+https://medium.com/@checko/merging-two-git-repositories-into-one-preserving-the-git-history-4e20d3fafa4e
+https://thoughts.t37.net/merging-2-different-git-repositories-without-losing-your-history-de7a06bba804
+https://blog.doismellburning.co.uk/merging-two-git-repositories/
+
+Other much more complicated variations.
+
+http://mbork.pl/2019-08-19_Transplanting_a_directory_to_another_Git_repository
+
+
+-------------------------------------------------------------------
+
+Install / update / upgrade.
+
+I could keep the version number around.
+    Include version numbers / branches of dependencies.
+    Update will grab any security updates for the installed version.
+    Upgrade   will upgrade   to a chosen later    different version.
+    Downgrade will downgrade to a chosen earlier  different version.
+
+Note that we are currently using the LuaJIT 2.1.0-beta3 branch of the
+main Luajit repo.  Everything else is on their master branches.
+
+Bootstrap -
+    bootstrap.sh or bootstrap.bat
+
+    Build the LuaJIT that comes with our source.  It "builds out-of-the
+    box on most systems" and has no dependencies, other than a C build system.
+
+    Or download a prebuilt LuaJIT from somewhere.
+
+  After toybox has been LuaJITized.
+
+    Build the LuaJIT that comes with our source.  It "builds out-of-the
+    box on most systems" and has no dependencies, other than a C build system.
+
+    Similar should apply to toybox, though it's our LuaJITized version.
+        Will need a specific miniconfig for this that doesn't include sledjchisl.
+
+    Or download a prebuilt toybox+LuaJIT from a SledjHamr package repo.
+
+Install -
+    install.lua
+
+        Will need a pre flight check if the dependencies are installed.
+        It checks if the system is built and has source.
+            Build it all.
+        Do the usual copy stuff to a directory thing.
+        Run "sledjchisl -install" in that directory.
+            Which does the usual "check health of system and fix up problems" thing, then quits instead of keep running.
+                The health check should include making sure our database creds exist / work.
+
+Update / upgrade / downgrade
+    install.lua -update
+    install.lua -upgrade
+    install.lua -downgrade
+
+    Check if we are a binary only, or a source install.
+        wget new binaries / git pull new source
+            Toybox has a wget in pending, otherwise it only has ftpget.
+            Git is standalone outside of the system, but if you are
+            running from source, you likely have standard build tools
+            like git.
+
+
+Yeah I hate things that have their own packaging system, for needing to
+step outside the operating systems packaging system, and adding to the too
+long list of stuff I have to deal with manually, and now I are one.  lol