From 3f67f0304f2138a38cf2695afec163b12db02d2c Mon Sep 17 00:00:00 2001 From: onefang Date: Mon, 16 Mar 2020 20:22:12 +1000 Subject: Finally add the sledjchisl C & Lua maanger, and friends. --- src/sledjchisl/NOTES.txt | 503 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 503 insertions(+) create mode 100644 src/sledjchisl/NOTES.txt (limited to 'src/sledjchisl/NOTES.txt') diff --git a/src/sledjchisl/NOTES.txt b/src/sledjchisl/NOTES.txt new file mode 100644 index 0000000..a5f0861 --- /dev/null +++ b/src/sledjchisl/NOTES.txt @@ -0,0 +1,503 @@ +I'm re-purposing this for SledjHamr https://sledjhamr.org/git/docs/index.html + +The general structure of SledjHamr is a bunch of servers talking to each +other via Internet (or just local) connections. One of them is a web +server for assets, world data, and inventory. + +Originally I didn't think using a web based world client was a good idea, +however it might be better to have one, for reasons. Now I need a web +management console that can do all the things the current tmux console +can, including OpenSim console and commands. Plus account management for +users. I can also use a web based Jabber / XMPP front end to chat, IM, +and group chatter, which would run in the normal viewers web browser. +This provides a doorway into putting SledjHamr stuff in existing viewers +without needing them to support it. So a web based viewer now makes more +sense, and also means we can get away with not needing a viewer at all. + +Toybox itself doesn't include a web server, and I don't think there is +one on the roadmap. So we have to use an external web server, which was +a design goal of SledjHamr in the first place, using existing mature +HTTP infrastructure, coz that's already solved problems for a bunch of +things that plague OS/SL to this day. Clear your cache! Pffft. + +So sledjchisl.c will be the "love world server", though initially it just +drives OpenSim_SC in tmux via tmux commands to send keys and read output. +Later it might run opensim_SC directly and use STDIN and STDOUT to do +everything. It'll also provide the text management front end that runs +in the left tmux panel of the first window, which is why it's based on +boxes in the first place. Later still it can take over opensim_SC +functions as I move them out of mono. + +We will need a text, web, and GUI version of this management front end. +Hmmm, maybe don't need a GUI version, GUI users can just run a terminal. + +After much research, FastCGI / FCGI seems to be the most portable way of +interfacing with existing web servers. FCGI protocol closes STDERR and +STDOUT, and uses STDIN as it's two way communications channel to the web +server, so our FCGI module can't be used as the text management front +end. This is probably a good idea to keep them seperate anyway, for +security, coz the web server is exposed to the world, the console isn't. + +Currently sledjchisl.c tests to see if it's running in tmux already, if +it isn't it starts up tmux runs itself into this new tmux, then exits. +So it could also test if it's running from FCGI, and switch to web mode, +then it'll need to find the tmuxed instance to send commands to it. +Either via nails connection, or sending tmux commands via shell. + +FCGI has methods of dealing with auth and templates. B-) + +So for now I think I'll have the text and web management front ends in +sledjchisl.c, and the love world server as well. I can split them up +later if I need to. + + + + +I has Apache 2.4.25-3+deb9u9 + MariaDB 10.1.44-MariaDB + + +https://gist.github.com/dermesser/e2f9b66457ae19ebd116 + Multithreaded example in C. + + +------------------------------------------------------------------- + +Apache doesn't seem to support FCGI filter role, so I might have to do +without. Might be better anyway. + + +"A Filter is similar in functionality to a Responder that takes a data +file as a parameter. The difference is that with a Filter, both the data +file and the Filter itself can be access controlled using the Web +server's access control mechanisms, while a Responder that takes the name +of a data file as a parameter must perform its own access control checks +on the data file." + + Which is fine, our access control checks will be "Is this database + defined user already logged on via our FCGI script?". We should have + total control over that. I was planning on using the FCGI auth + mechanism anyway. + + +RESPONDER + web server sends FCGI_PARAMS + CONTENT_LENGTH + web server sends input body FCGI_STDIN + fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR + it has to finish reading FCGI_PARAMS first + fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE) + + +FILTER + filtered file has last modified time + web server sets FCGI_DATA_LAST_MOD accordingly + web server sends FCGI_PARAMS + CONTENT_LENGTH FCGI_DATA_LAST_MOD FCGI_DATA_LENGTH + web server sends input body FCGI_STDIN + web servers sends file over FCGI_DATA + fcgi app can ignore FCGI_DATA and use it's own cached copy based on FCGI_DATA_LAST_MOD + fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR + it has to finish reading FCGI_STDIN first, but not FCGI_DATA + fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE) + + +Soooo, FILTER might be slower anyway if we are caching the filtered file, +or mmapping it, coz filter has to start sending the filtered file, even +if it's to be served from cache. Plus no need to wait for FCGI_STDIN +before spewing it out. + + +Last update time for parameters, plus an update frequency. Once a minute. + + NOTE - SSI is a bit more complex than what I'm currently using. + https://en.wikipedia.org/wiki/Server_Side_Includes + + + + +. + + + + + + + + + + + + + + + https://www.w3.org/Jigsaw/Doc/User/SSI.html + Adds lots of others, including Java stuff. + Mine + + + + +------------------------------------------------------------------- + +Account creation process in the database. + +Apart from the usual input validation of things... + + +OpenSim/Server/Handlers/UserAccounts/UserAccountServerPostHandler.cs + byte[] CreateUser(Dictionary request) + Looks like their built in web front end, perhaps what is triggered by the console? + createdUserAccount + = ((UserAccountService)m_UserAccountService).CreateUser(scopeID, principalID, firstName, lastName, password, email, model); + +OpenSim/opensim-SC/OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs + An XML RPC interface to - + private UserAccount CreateUser(UUID scopeID, string firstName, string lastName, string password, string email) + account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email); + if (userAccountService.StoreUserAccount(account)) + success = authenticationService.SetPassword(account.PrincipalID, password) + gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0)); + success = inventoryService.CreateUserInventory(account.PrincipalID); + +OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs + Looks like the console command handler. + create user [ [ [ [ [ []]]]]] - Create a new user + protected void HandleCreateUser(string module, string[] cmdparams) + Gathers console arguments, or prompts for them. + CreateUser(UUID.Zero, principalId, firstName, lastName, password, email, model); + public UserAccount CreateUser(UUID scopeID, UUID principalID, string firstName, string lastName, string password, string email, string model = "") + Looks almost identical to the OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs one above, but they add - + CreateDefaultAppearanceEntries(account.PrincipalID) + + + +account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email); + OpenSim/opensim-SC/OpenSim/Services/Interfaces/IUserAccountService.cs + public UserAccount(UUID scopeID, UUID principalID, string firstName, string lastName, string email) + Just holds the data in memory, in a dictionary I think. + OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs + public bool StoreUserAccount(UserAccount data) + Stuffs the data into a new UserAccountData() + m_Database.Store(d) + As far as I can tell, just dumps this data into the UserAccounts table - + FirstName, LastName, PrincipleID, ScopeID, Email, Created, UserLevel, UserFlags, UserTitle + PrincipleID is their randomly generated with no thought to collisions UUID. + ScopeID is 00000000-0000-0000-0000-000000000000 + Userlevel is 0 for most, -1 for Waki, determines if they can log on. Also higher for gods and things. + UserFlags, I think the only one is "64 god can login to this account using gods password. + UserTitle might default to "Local", or be configurable / and editable. + something something URL encoded "ServiceURLs" mumble + HomeURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GatekeeperURI= InventoryServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f AssetServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f ProfileServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f FriendsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f IMServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GroupsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f + Though most are either NULL, empty, or - + HomeURI= GatekeeperURI= InventoryServerURI= AssetServerURI= + Doesn't metion "active", which is always equal to 1 I guess. + + + +success = authenticationService.SetPassword(account.PrincipalID, password) + OpenSim/Services/AuthenticationService/AuthenticationServiceBase.cs + stores password details in "auth" table - + UUID + passwordSalt = Util.Md5Hash(UUID.Random().ToString()); + passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt); + accountType = "UserAccount"; + webLoginKey = UUID.Zero.ToString(); + + + +gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0)); + OpenSim/Services/UserAccountService/GridUserService.cs + Stores in database table GridUser + HomeRegionID, HomePosition, HomeLookAt + The other fields in that table - + UserID, LastRegionID, LastPosition, LastLookAt, Online (true or false), Login (timestamp or 0), Logout (timestamp or 0). + + + +success = inventoryService.CreateUserInventory(account.PrincipalID); + OpenSim/Services/InventoryService/XInventoryService.cs + Create a bunch of folders in the users inventory, of specific types. + rootFolder = ConvertToOpenSim(CreateFolder(principalID, UUID.Zero, (int)FolderType.Root, InventoryFolderBase.ROOT_FOLDER_NAME)); + XInventoryFolder[] sysFolders = GetSystemFolders(principalID, rootFolder.ID) + if (!Array.Exists(sysFolders, delegate(XInventoryFolder f) { if (f.type == (int)FolderType.Animation) return true; return false; })) + CreateFolder(principalID, rootFolder.ID, (int)FolderType.Animation, "Animations"); + FolderType.BodyPart, "Body Parts" + XInventoryFolder folder = CreateFolder(principalID, rootFolder.ID, (int)FolderType.CallingCard, "Calling Cards"); + folder = CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "Friends") + CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "All"); + FolderType.Clothing, "Clothing" + FolderType.CurrentOutfit, "Current Outfit" + FolderType.Favorites, "Favorites" + FolderType.Gesture, "Gestures") + FolderType.Landmark, "Landmarks" + FolderType.LostAndFound, "Lost And Found" + FolderType.Notecard, "Notecards" + FolderType.Object, "Objects" + FolderType.Snapshot, "Photo Album" + FolderType.LSLText, "Scripts" + FolderType.Sound, "Sounds" + FolderType.Texture, "Textures" + FolderType.Trash, "Trash" + + Stores in database inventoryFolders ???? + folderName, type, version = 1, folderID = UUID.Random(), agentID = principalID, parentFolderID = parentID + + + + +CreateDefaultAppearanceEntries(account.PrincipalID) + OpenSim/Services/UserAccountService/UserAccountService.cs + protected void CreateDefaultAppearanceEntries(UUID principalID) + Creates a bunch of "Default *" body parts and clothes, Ruth 1.0, links them in Inventories current outfit folder. + Creates a AvatarWearable[] and puts them all in it. + AvatarAppearance ap = new AvatarAppearance(); + ap.SetWearable(i, wearables[i]); + m_AvatarService.SetAppearance(principalID, ap); + + + + + +UserAccounts table - + UserFlags 64 is "allow gods to log in as me" + 0xf00 is membershipType, unles there's a title. Only sent to viewers I think. + 32 is Minors for estate banning purposes. + 4 is Anonymous for estate banning purposes. + 1 is AllowPublish in profile, but userprofile has this as separate field. + 2 is MaturePublish in profile, but userprofile has this as separate field. +Presence table - + UserID varchar(255) + RegionID char(36) + SessionID char(36) + SecureSessionID char(36) + LastSeen timestamp +tokens table (I think this is actually used for something) - + UUID char(36) + token varchar(255) current example looks like a UUID. + validity datetime +userdata (empty, can't find any actual usage in the source code, part of profiles) - + UserId char(36) primary index + TagId varchar(64) primary index + DataKey varchar(255) + DataVal varchar(255) +auth.webLoginKey seems to be some sort of passwordy type thing, though perhaps not actually hashed, rarely used, none of IG members have one. + + +PLAN- +. username +. password +. create login + +.check if it's a proper two word name +.login -> check if it's an existing account, get their UUID. + create toke_n_munchie + write session record + +create -> new user + create new UUID + check if it's an existing UUID + dbCount(, "UserAccounts", "PrincipleID='new-UUID'") + loop until we get a new one + create toke_n_munchie + write session record + + + Create -> + (wait a few seconds before showing this page) +. email +. email again +. password again +. DoB +. accept terms of service +. claim to be an adult +. confirm / cancel + + New user + UserAccounts.FirstName = ??? + UserAccounts.LastName = ??? + UserAccounts.Email = ??? + UserAccounts.Created = timestamp + UserAccounts.PrincipleID = randomly generate UUID, but check for collisions with other accounts. + It's a UNIQUE KEY. + UserAccounts.ScopeID = 00000000-0000-0000-0000-000000000000 + UserAccounts.Userlevel = -200 + UserAccounts.UserFlags = 64 + UserAccounts.UserTitle = newbie + UserAccounts.ServiceURLs = "" + UserAccounts.active = 0 + + auth.UUID = UserAccounts.PrincipleID + It's a PRIMARY KEY. + auth.passwordSalt = Util.Md5Hash(UUID.Random().ToString()) + auth.passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt) + auth.accountType = "UserAccount" + auth.webLoginKey (varchar(255)) = "00000000-0000-0000-0000-000000000000" + + userdata.UserId = UserAccounts.PrincipleID + userdata.TagId = "account creation data" + It's a UNIQUE KEY + userdata.DataKey = "DoB" + userdata.DataVal = ??? + + userdata.UserId = UserAccounts.PrincipleID + userdata.TagId = "account creation data" + userdata.DataKey = "timezone" + userdata.DataVal = ??? + + userdata.UserId = UserAccounts.PrincipleID + userdata.TagId = "account creation data" + userdata.DataKey = "Terms of service" + userdata.DataVal = "True" + + userdata.UserId = UserAccounts.PrincipleID + userdata.TagId = "account creation data" + userdata.DataKey = "claims to be an adult" + userdata.DataVal = "True" + + + Validated via email + (wait a few seconds before showing this page) + UserAccounts.Userlevel = -100 + UserAccounts.UserTitle = validated + + + Vouched for + userdata.UserId = UserAccounts.PrincipleID + userdata.TagId = "vouches" + userdata.DataKey = UUID of voucher + userdata.DataVal = timestamp of vouching + + UserAccounts.Userlevel = -50 + UserAccounts.UserTitle = vouched for + + + Admin approved + GridUser.UserID = UserAccounts.PrincipleID + It's a PRIMARY KEY. + GridUser.HomeRegionID = ??? + GridUser.HomePosition = ??? + GridUser.HomeLookAt = ??? + GridUser.LastRegionID = ??? + GridUser.LastPosition = ??? + GridUser.LastLookAt = ??? + GridUser.Online = False + GridUser.Login = 0 + GridUser.Logout = 0 + + UserAccounts.active = 1 + UserAccounts.Userlevel = 1 + UserAccounts.UserTitle = Member / Local / whatever + + Load the default IAR. + + +------------------------------------------------------------------- + +https://project-awesome.org/aleksandar-todorovic/awesome-c + A curated list of C good stuff. + +https://wolkykim.github.io/qdecoder/ + CGI library made by the qlibc guy, does support FCGI. + Might be a wrapper around the fcgi_stdio stuff I'm already using? + + +https://danielmiessler.com/study/http/ + A Security-focused HTTP Primer + Nothing much new except to say this about the Referer header - + "should not be used to make security decisions as it is controlled by the client" + Though others tell us to do precisely that. lol + + +------------------------------------------------------------------- + +apt install libmariadbclient-dev libapache2-mod-fcgid + +------------------------------------------------------------------- + +Merge it into OpenSim-SC. + + Complication - I had already added the boxes + early sledjchisl.c to opensim-SC. + I may have to revert that lot, only a few minor commits, which are already part of the main boxes. + d9c772712e27c8e25fab0d17555a9bc11017a125 + d4ea3e50173df1ad646bdb7dc802f5d320b7e511 + 10ed36b3452ce6373175112716b043047dc896a9 + 2f66c46e7ce18d60cd1f565880fb7762b3399ccb + 34c5ee4c2a489a506e93d5b303fbc80b263747f0 is the commit that added it. + f9bfa831d1ccaa973c42042584510e1a724ddaef the one before it. + It's been pushed up to the sledjhamr.org repo. + + git revert d9c772712e27c8e25fab0d17555a9bc11017a125 d4ea3e50173df1ad646bdb7dc802f5d320b7e511 10ed36b3452ce6373175112716b043047dc896a9 2f66c46e7ce18d60cd1f565880fb7762b3399ccb 34c5ee4c2a489a506e93d5b303fbc80b263747f0 + git remote add -f boxes ~/MyLinux_3/TOYBOX/sledjchisl + git merge --allow-unrelated-histories boxes/master + git remote rm boxes + + +https://medium.com/altcampus/how-to-merge-two-or-multiple-git-repositories-into-one-9f8a5209913f +https://medium.com/@checko/merging-two-git-repositories-into-one-preserving-the-git-history-4e20d3fafa4e +https://thoughts.t37.net/merging-2-different-git-repositories-without-losing-your-history-de7a06bba804 +https://blog.doismellburning.co.uk/merging-two-git-repositories/ + +Other much more complicated variations. + +http://mbork.pl/2019-08-19_Transplanting_a_directory_to_another_Git_repository + + +------------------------------------------------------------------- + +Install / update / upgrade. + +I could keep the version number around. + Include version numbers / branches of dependencies. + Update will grab any security updates for the installed version. + Upgrade will upgrade to a chosen later different version. + Downgrade will downgrade to a chosen earlier different version. + +Note that we are currently using the LuaJIT 2.1.0-beta3 branch of the +main Luajit repo. Everything else is on their master branches. + +Bootstrap - + bootstrap.sh or bootstrap.bat + + Build the LuaJIT that comes with our source. It "builds out-of-the + box on most systems" and has no dependencies, other than a C build system. + + Or download a prebuilt LuaJIT from somewhere. + + After toybox has been LuaJITized. + + Build the LuaJIT that comes with our source. It "builds out-of-the + box on most systems" and has no dependencies, other than a C build system. + + Similar should apply to toybox, though it's our LuaJITized version. + Will need a specific miniconfig for this that doesn't include sledjchisl. + + Or download a prebuilt toybox+LuaJIT from a SledjHamr package repo. + +Install - + install.lua + + Will need a pre flight check if the dependencies are installed. + It checks if the system is built and has source. + Build it all. + Do the usual copy stuff to a directory thing. + Run "sledjchisl -install" in that directory. + Which does the usual "check health of system and fix up problems" thing, then quits instead of keep running. + The health check should include making sure our database creds exist / work. + +Update / upgrade / downgrade + install.lua -update + install.lua -upgrade + install.lua -downgrade + + Check if we are a binary only, or a source install. + wget new binaries / git pull new source + Toybox has a wget in pending, otherwise it only has ftpget. + Git is standalone outside of the system, but if you are + running from source, you likely have standard build tools + like git. + + +Yeah I hate things that have their own packaging system, for needing to +step outside the operating systems packaging system, and adding to the too +long list of stuff I have to deal with manually, and now I are one. lol -- cgit v1.1