From b2750deb586644a8052b51b1cf60de731462861c Mon Sep 17 00:00:00 2001 From: onefang Date: Sun, 19 Apr 2020 14:11:16 +1000 Subject: Move NOTES.txt, and write some more. --- src/sledjchisl/NOTES.txt | 479 ----------------------------------------------- 1 file changed, 479 deletions(-) delete mode 100644 src/sledjchisl/NOTES.txt (limited to 'src/sledjchisl') diff --git a/src/sledjchisl/NOTES.txt b/src/sledjchisl/NOTES.txt deleted file mode 100644 index e80b8d9..0000000 --- a/src/sledjchisl/NOTES.txt +++ /dev/null @@ -1,479 +0,0 @@ -I'm re-purposing this for SledjHamr https://sledjhamr.org/git/docs/index.html - -The general structure of SledjHamr is a bunch of servers talking to each -other via Internet (or just local) connections. One of them is a web -server for assets, world data, and inventory. - -Originally I didn't think using a web based world client was a good idea, -however it might be better to have one, for reasons. Now I need a web -management console that can do all the things the current tmux console -can, including OpenSim console and commands. Plus account management for -users. I can also use a web based Jabber / XMPP front end to chat, IM, -and group chatter, which would run in the normal viewers web browser. -This provides a doorway into putting SledjHamr stuff in existing viewers -without needing them to support it. So a web based viewer now makes more -sense, and also means we can get away with not needing a viewer at all. - -Toybox itself doesn't include a web server, and I don't think there is -one on the roadmap. So we have to use an external web server, which was -a design goal of SledjHamr in the first place, using existing mature -HTTP infrastructure, coz that's already solved problems for a bunch of -things that plague OS/SL to this day. Clear your cache! Pffft. - -So sledjchisl.c will be the "love world server", though initially it just -drives OpenSim_SC in tmux via tmux commands to send keys and read output. -Later it might run opensim_SC directly and use STDIN and STDOUT to do -everything. It'll also provide the text management front end that runs -in the left tmux panel of the first window, which is why it's based on -boxes in the first place. Later still it can take over opensim_SC -functions as I move them out of mono. - -We will need a text, web, and GUI version of this management front end. -Hmmm, maybe don't need a GUI version, GUI users can just run a terminal. - -After much research, FastCGI / FCGI seems to be the most portable way of -interfacing with existing web servers. FCGI protocol closes STDERR and -STDOUT, and uses STDIN as it's two way communications channel to the web -server, so our FCGI module can't be used as the text management front -end. This is probably a good idea to keep them seperate anyway, for -security, coz the web server is exposed to the world, the console isn't. - -Currently sledjchisl.c tests to see if it's running in tmux already, if -it isn't it starts up tmux runs itself into this new tmux, then exits. -So it could also test if it's running from FCGI, and switch to web mode, -then it'll need to find the tmuxed instance to send commands to it. -Either via nails connection, or sending tmux commands via shell. - -FCGI has methods of dealing with auth and templates. B-) - -So for now I think I'll have the text and web management front ends in -sledjchisl.c, and the love world server as well. I can split them up -later if I need to. - - - - -I has Apache 2.4.25-3+deb9u9 - MariaDB 10.1.44-MariaDB - - -https://gist.github.com/dermesser/e2f9b66457ae19ebd116 - Multithreaded example in C. - - -------------------------------------------------------------------- - -Apache doesn't seem to support FCGI filter role, so I might have to do -without. Might be better anyway. - - -"A Filter is similar in functionality to a Responder that takes a data -file as a parameter. The difference is that with a Filter, both the data -file and the Filter itself can be access controlled using the Web -server's access control mechanisms, while a Responder that takes the name -of a data file as a parameter must perform its own access control checks -on the data file." - - Which is fine, our access control checks will be "Is this database - defined user already logged on via our FCGI script?". We should have - total control over that. I was planning on using the FCGI auth - mechanism anyway. - - -RESPONDER - web server sends FCGI_PARAMS - CONTENT_LENGTH - web server sends input body FCGI_STDIN - fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR - it has to finish reading FCGI_PARAMS first - fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE) - - -FILTER - filtered file has last modified time - web server sets FCGI_DATA_LAST_MOD accordingly - web server sends FCGI_PARAMS - CONTENT_LENGTH FCGI_DATA_LAST_MOD FCGI_DATA_LENGTH - web server sends input body FCGI_STDIN - web servers sends file over FCGI_DATA - fcgi app can ignore FCGI_DATA and use it's own cached copy based on FCGI_DATA_LAST_MOD - fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR - it has to finish reading FCGI_STDIN first, but not FCGI_DATA - fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE) - - -Soooo, FILTER might be slower anyway if we are caching the filtered file, -or mmapping it, coz filter has to start sending the filtered file, even -if it's to be served from cache. Plus no need to wait for FCGI_STDIN -before spewing it out. - - -Last update time for parameters, plus an update frequency. Once a minute. - - NOTE - SSI is a bit more complex than what I'm currently using. - https://en.wikipedia.org/wiki/Server_Side_Includes - - - - -. - - - - - - - - - - - - - - - https://www.w3.org/Jigsaw/Doc/User/SSI.html - Adds lots of others, including Java stuff. - Mine - - - - -------------------------------------------------------------------- - -Account creation process in the database. - -Apart from the usual input validation of things... - - -OpenSim/Server/Handlers/UserAccounts/UserAccountServerPostHandler.cs - byte[] CreateUser(Dictionary request) - Looks like their built in web front end, perhaps what is triggered by the console? - createdUserAccount - = ((UserAccountService)m_UserAccountService).CreateUser(scopeID, principalID, firstName, lastName, password, email, model); - -OpenSim/opensim-SC/OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs - An XML RPC interface to - - private UserAccount CreateUser(UUID scopeID, string firstName, string lastName, string password, string email) - account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email); - if (userAccountService.StoreUserAccount(account)) - success = authenticationService.SetPassword(account.PrincipalID, password) - gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0)); - success = inventoryService.CreateUserInventory(account.PrincipalID); - -OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs - Looks like the console command handler. - create user [ [ [ [ [ []]]]]] - Create a new user - protected void HandleCreateUser(string module, string[] cmdparams) - Gathers console arguments, or prompts for them. - CreateUser(UUID.Zero, principalId, firstName, lastName, password, email, model); - public UserAccount CreateUser(UUID scopeID, UUID principalID, string firstName, string lastName, string password, string email, string model = "") - Looks almost identical to the OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs one above, but they add - - CreateDefaultAppearanceEntries(account.PrincipalID) - - - -account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email); - OpenSim/opensim-SC/OpenSim/Services/Interfaces/IUserAccountService.cs - public UserAccount(UUID scopeID, UUID principalID, string firstName, string lastName, string email) - Just holds the data in memory, in a dictionary I think. - OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs - public bool StoreUserAccount(UserAccount data) - Stuffs the data into a new UserAccountData() - m_Database.Store(d) - As far as I can tell, just dumps this data into the UserAccounts table - - FirstName, LastName, PrincipleID, ScopeID, Email, Created, UserLevel, UserFlags, UserTitle - PrincipleID is their randomly generated with no thought to collisions UUID. - ScopeID is 00000000-0000-0000-0000-000000000000 - Userlevel is 0 for most, -1 for Waki, determines if they can log on. Also higher for gods and things. - UserFlags, I think the only one is "64 god can login to this account using gods password. - UserTitle might default to "Local", or be configurable / and editable. - something something URL encoded "ServiceURLs" mumble - HomeURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GatekeeperURI= InventoryServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f AssetServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f ProfileServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f FriendsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f IMServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GroupsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f - Though most are either NULL, empty, or - - HomeURI= GatekeeperURI= InventoryServerURI= AssetServerURI= - Doesn't metion "active", which is always equal to 1 I guess. - - - -success = authenticationService.SetPassword(account.PrincipalID, password) - OpenSim/Services/AuthenticationService/AuthenticationServiceBase.cs - stores password details in "auth" table - - UUID - passwordSalt = Util.Md5Hash(UUID.Random().ToString()); - passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt); - accountType = "UserAccount"; - webLoginKey = UUID.Zero.ToString(); - - - -gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0)); - OpenSim/Services/UserAccountService/GridUserService.cs - Stores in database table GridUser - HomeRegionID, HomePosition, HomeLookAt - The other fields in that table - - UserID, LastRegionID, LastPosition, LastLookAt, Online (true or false), Login (timestamp or 0), Logout (timestamp or 0). - - - -success = inventoryService.CreateUserInventory(account.PrincipalID); - OpenSim/Services/InventoryService/XInventoryService.cs - Create a bunch of folders in the users inventory, of specific types. - rootFolder = ConvertToOpenSim(CreateFolder(principalID, UUID.Zero, (int)FolderType.Root, InventoryFolderBase.ROOT_FOLDER_NAME)); - XInventoryFolder[] sysFolders = GetSystemFolders(principalID, rootFolder.ID) - if (!Array.Exists(sysFolders, delegate(XInventoryFolder f) { if (f.type == (int)FolderType.Animation) return true; return false; })) - CreateFolder(principalID, rootFolder.ID, (int)FolderType.Animation, "Animations"); - FolderType.BodyPart, "Body Parts" - XInventoryFolder folder = CreateFolder(principalID, rootFolder.ID, (int)FolderType.CallingCard, "Calling Cards"); - folder = CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "Friends") - CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "All"); - FolderType.Clothing, "Clothing" - FolderType.CurrentOutfit, "Current Outfit" - FolderType.Favorites, "Favorites" - FolderType.Gesture, "Gestures") - FolderType.Landmark, "Landmarks" - FolderType.LostAndFound, "Lost And Found" - FolderType.Notecard, "Notecards" - FolderType.Object, "Objects" - FolderType.Snapshot, "Photo Album" - FolderType.LSLText, "Scripts" - FolderType.Sound, "Sounds" - FolderType.Texture, "Textures" - FolderType.Trash, "Trash" - - Stores in database inventoryFolders ???? - folderName, type, version = 1, folderID = UUID.Random(), agentID = principalID, parentFolderID = parentID - - - - -CreateDefaultAppearanceEntries(account.PrincipalID) - OpenSim/Services/UserAccountService/UserAccountService.cs - protected void CreateDefaultAppearanceEntries(UUID principalID) - Creates a bunch of "Default *" body parts and clothes, Ruth 1.0, links them in Inventories current outfit folder. - Creates a AvatarWearable[] and puts them all in it. - AvatarAppearance ap = new AvatarAppearance(); - ap.SetWearable(i, wearables[i]); - m_AvatarService.SetAppearance(principalID, ap); - - - - - -UserAccounts table - - UserFlags 64 is "allow gods to log in as me" - 0xf00 is membershipType, unles there's a title. Only sent to viewers I think. - 32 is Minors for estate banning purposes. - 4 is Anonymous for estate banning purposes. - 1 is AllowPublish in profile, but userprofile has this as separate field. - 2 is MaturePublish in profile, but userprofile has this as separate field. -Presence table - - UserID varchar(255) - RegionID char(36) - SessionID char(36) - SecureSessionID char(36) - LastSeen timestamp -tokens table (I think this is actually used for something) - - UUID char(36) - token varchar(255) current example looks like a UUID. - validity datetime -userdata (empty, can't find any actual usage in the source code, part of profiles) - - UserId char(36) primary index - TagId varchar(64) primary index - DataKey varchar(255) - DataVal varchar(255) -auth.webLoginKey seems to be some sort of passwordy type thing, though perhaps not actually hashed, rarely used, none of IG members have one. - - -PLAN- -. username -. password -. create login - -.check if it's a proper two word name -.login -> check if it's an existing account, get their UUID. - create toke_n_munchie - write session record - -create -> new user - create new UUID - check if it's an existing UUID - dbCount(, "UserAccounts", "PrincipleID='new-UUID'") - loop until we get a new one - create toke_n_munchie - write session record - - - Create -> - (wait a few seconds before showing this page) -. email -. email again -. password again -. DoB -. accept terms of service -. claim to be an adult -. confirm / cancel - - New user - UserAccounts.FirstName = ??? - UserAccounts.LastName = ??? - UserAccounts.Email = ??? - UserAccounts.Created = timestamp - UserAccounts.PrincipleID = randomly generate UUID, but check for collisions with other accounts. - It's a UNIQUE KEY. - UserAccounts.ScopeID = 00000000-0000-0000-0000-000000000000 - UserAccounts.Userlevel = -200 - UserAccounts.UserFlags = 64 - UserAccounts.UserTitle = newbie - UserAccounts.ServiceURLs = "" - UserAccounts.active = 0 - - auth.UUID = UserAccounts.PrincipleID - It's a PRIMARY KEY. - auth.passwordSalt = Util.Md5Hash(UUID.Random().ToString()) - auth.passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt) - auth.accountType = "UserAccount" - auth.webLoginKey (varchar(255)) = "00000000-0000-0000-0000-000000000000" - - userdata.UserId = UserAccounts.PrincipleID - userdata.TagId = "account creation data" - It's a UNIQUE KEY - userdata.DataKey = "DoB" - userdata.DataVal = ??? - - userdata.UserId = UserAccounts.PrincipleID - userdata.TagId = "account creation data" - userdata.DataKey = "timezone" - userdata.DataVal = ??? - - userdata.UserId = UserAccounts.PrincipleID - userdata.TagId = "account creation data" - userdata.DataKey = "Terms of service" - userdata.DataVal = "True" - - userdata.UserId = UserAccounts.PrincipleID - userdata.TagId = "account creation data" - userdata.DataKey = "claims to be an adult" - userdata.DataVal = "True" - - - Validated via email - (wait a few seconds before showing this page) - UserAccounts.Userlevel = -100 - UserAccounts.UserTitle = validated - - - Vouched for - userdata.UserId = UserAccounts.PrincipleID - userdata.TagId = "vouches" - userdata.DataKey = UUID of voucher - userdata.DataVal = timestamp of vouching - - UserAccounts.Userlevel = -50 - UserAccounts.UserTitle = vouched for - - - Admin approved - GridUser.UserID = UserAccounts.PrincipleID - It's a PRIMARY KEY. - GridUser.HomeRegionID = ??? - GridUser.HomePosition = ??? - GridUser.HomeLookAt = ??? - GridUser.LastRegionID = ??? - GridUser.LastPosition = ??? - GridUser.LastLookAt = ??? - GridUser.Online = False - GridUser.Login = 0 - GridUser.Logout = 0 - - UserAccounts.active = 1 - UserAccounts.Userlevel = 1 - UserAccounts.UserTitle = Member / Local / whatever - - Load the default IAR. - - -------------------------------------------------------------------- - -https://project-awesome.org/aleksandar-todorovic/awesome-c - A curated list of C good stuff. - -https://wolkykim.github.io/qdecoder/ - CGI library made by the qlibc guy, does support FCGI. - Might be a wrapper around the fcgi_stdio stuff I'm already using? - - -https://danielmiessler.com/study/http/ - A Security-focused HTTP Primer - Nothing much new except to say this about the Referer header - - "should not be used to make security decisions as it is controlled by the client" - Though others tell us to do precisely that. lol - - -------------------------------------------------------------------- - -apt install libmariadbclient-dev libapache2-mod-fcgid libssl1.0-dev uuid-dev spawn-fcgi - -BTW, everything is some BSD licence variation, except MariaDB, which is -some combination of GPLv2, 2+, 3, and BSD variants. With FOSS License -Exception, which seems to mean "Oracles GPL isn't viral". Though my -license is "fuck you viral GPL, that's not your decision to make for my -fucking code" BSD. - -------------------------------------------------------------------- - -Install / update / upgrade. - -I could keep the version number around. - Include version numbers / branches of dependencies. - Update will grab any security updates for the installed version. - Upgrade will upgrade to a chosen later different version. - Downgrade will downgrade to a chosen earlier different version. - -Note that we are currently using the LuaJIT 2.1.0-beta3 branch of the -main Luajit repo. Everything else is on their master branches. - -Bootstrap - - bootstrap.sh or bootstrap.bat - - Build the LuaJIT that comes with our source. It "builds out-of-the - box on most systems" and has no dependencies, other than a C build system. - - Or download a prebuilt LuaJIT from somewhere. - - After toybox has been LuaJITized. - - Build the LuaJIT that comes with our source. It "builds out-of-the - box on most systems" and has no dependencies, other than a C build system. - - Similar should apply to toybox, though it's our LuaJITized version. - Will need a specific miniconfig for this that doesn't include sledjchisl. - - Or download a prebuilt toybox+LuaJIT from a SledjHamr package repo. - -Install - - install.lua - - Will need a pre flight check if the dependencies are installed. - It checks if the system is built and has source. - Build it all. - Do the usual copy stuff to a directory thing. - Run "sledjchisl -install" in that directory. - Which does the usual "check health of system and fix up problems" thing, then quits instead of keep running. - The health check should include making sure our database creds exist / work. - -Update / upgrade / downgrade - install.lua -update - install.lua -upgrade - install.lua -downgrade - - Check if we are a binary only, or a source install. - wget new binaries / git pull new source - Toybox has a wget in pending, otherwise it only has ftpget. - Git is standalone outside of the system, but if you are - running from source, you likely have standard build tools - like git. - - -Yeah I hate things that have their own packaging system, for needing to -step outside the operating systems packaging system, and adding to the too -long list of stuff I have to deal with manually, and now I are one. lol -- cgit v1.1