From b2750deb586644a8052b51b1cf60de731462861c Mon Sep 17 00:00:00 2001 From: onefang Date: Sun, 19 Apr 2020 14:11:16 +1000 Subject: Move NOTES.txt, and write some more. --- src/NOTES.txt | 651 +++++++++++++++++++++++++++++++++++++++++++++++ src/sledjchisl/NOTES.txt | 479 ---------------------------------- 2 files changed, 651 insertions(+), 479 deletions(-) create mode 100644 src/NOTES.txt delete mode 100644 src/sledjchisl/NOTES.txt (limited to 'src') diff --git a/src/NOTES.txt b/src/NOTES.txt new file mode 100644 index 0000000..93ed815 --- /dev/null +++ b/src/NOTES.txt @@ -0,0 +1,651 @@ +I'm re-purposing this for SledjHamr https://sledjhamr.org/git/docs/index.html + +The general structure of SledjHamr is a bunch of servers talking to each +other via Internet (or just local) connections. One of them is a web +server for assets, world data, and inventory. + +Originally I didn't think using a web based world client was a good idea, +however it might be better to have one, for reasons. Now I need a web +management console that can do all the things the current tmux console +can, including OpenSim console and commands. Plus account management for +users. I can also use a web based Jabber / XMPP front end to chat, IM, +and group chatter, which would run in the normal viewers web browser. +This provides a doorway into putting SledjHamr stuff in existing viewers +without needing them to support it. So a web based viewer now makes more +sense, and also means we can get away with not needing a viewer at all. + +Toybox itself doesn't include a web server, and I don't think there is +one on the roadmap. So we have to use an external web server, which was +a design goal of SledjHamr in the first place, using existing mature +HTTP infrastructure, coz that's already solved problems for a bunch of +things that plague OS/SL to this day. Clear your cache! Pffft. + +So sledjchisl.c will be the "love world server", though initially it just +drives OpenSim_SC in tmux via tmux commands to send keys and read output. +Later it might run opensim_SC directly and use STDIN and STDOUT to do +everything. It'll also provide the text management front end that runs +in the left tmux panel of the first window, which is why it's based on +boxes in the first place. Later still it can take over opensim_SC +functions as I move them out of mono. + +We will need a text, web, and GUI version of this management front end. +Hmmm, maybe don't need a GUI version, GUI users can just run a terminal. + +After much research, FastCGI / FCGI seems to be the most portable way of +interfacing with existing web servers. FCGI protocol closes STDERR and +STDOUT, and uses STDIN as it's two way communications channel to the web +server, so our FCGI module can't be used as the text management front +end. This is probably a good idea to keep them seperate anyway, for +security, coz the web server is exposed to the world, the console isn't. + +Currently sledjchisl.c tests to see if it's running in tmux already, if +it isn't it starts up tmux runs itself into this new tmux, then exits. +So it could also test if it's running from FCGI, and switch to web mode, +then it'll need to find the tmuxed instance to send commands to it. +Either via nails connection, or sending tmux commands via shell. + +FCGI has methods of dealing with auth and templates. B-) + +So for now I think I'll have the text and web management front ends in +sledjchisl.c, and the love world server as well. I can split them up +later if I need to. + + + + +I has Apache 2.4.25-3+deb9u9 + MariaDB 10.1.44-MariaDB + + +https://gist.github.com/dermesser/e2f9b66457ae19ebd116 + Multithreaded example in C. + + +------------------------------------------------------------------- + +Apache doesn't seem to support FCGI filter role, so I might have to do +without. Might be better anyway. + + +"A Filter is similar in functionality to a Responder that takes a data +file as a parameter. The difference is that with a Filter, both the data +file and the Filter itself can be access controlled using the Web +server's access control mechanisms, while a Responder that takes the name +of a data file as a parameter must perform its own access control checks +on the data file." + + Which is fine, our access control checks will be "Is this database + defined user already logged on via our FCGI script?". We should have + total control over that. I was planning on using the FCGI auth + mechanism anyway. + + +RESPONDER + web server sends FCGI_PARAMS + CONTENT_LENGTH + web server sends input body FCGI_STDIN + fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR + it has to finish reading FCGI_PARAMS first + fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE) + + +FILTER + filtered file has last modified time + web server sets FCGI_DATA_LAST_MOD accordingly + web server sends FCGI_PARAMS + CONTENT_LENGTH FCGI_DATA_LAST_MOD FCGI_DATA_LENGTH + web server sends input body FCGI_STDIN + web servers sends file over FCGI_DATA + fcgi app can ignore FCGI_DATA and use it's own cached copy based on FCGI_DATA_LAST_MOD + fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR + it has to finish reading FCGI_STDIN first, but not FCGI_DATA + fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE) + + +Soooo, FILTER might be slower anyway if we are caching the filtered file, +or mmapping it, coz filter has to start sending the filtered file, even +if it's to be served from cache. Plus no need to wait for FCGI_STDIN +before spewing it out. + + +Last update time for parameters, plus an update frequency. Once a minute. + + NOTE - SSI is a bit more complex than what I'm currently using. + https://en.wikipedia.org/wiki/Server_Side_Includes + + + + +. + + + + + + + + + + + + + + + https://www.w3.org/Jigsaw/Doc/User/SSI.html + Adds lots of others, including Java stuff. + Mine + + + + +------------------------------------------------------------------- + +Account creation process in the database. + +Apart from the usual input validation of things... + + +OpenSim/Server/Handlers/UserAccounts/UserAccountServerPostHandler.cs + byte[] CreateUser(Dictionary request) + Looks like their built in web front end, perhaps what is triggered by the console? + createdUserAccount + = ((UserAccountService)m_UserAccountService).CreateUser(scopeID, principalID, firstName, lastName, password, email, model); + +OpenSim/opensim-SC/OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs + An XML RPC interface to - + private UserAccount CreateUser(UUID scopeID, string firstName, string lastName, string password, string email) + account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email); + if (userAccountService.StoreUserAccount(account)) + success = authenticationService.SetPassword(account.PrincipalID, password) + gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0)); + success = inventoryService.CreateUserInventory(account.PrincipalID); + +OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs + Looks like the console command handler. + create user [ [ [ [ [ []]]]]] - Create a new user + protected void HandleCreateUser(string module, string[] cmdparams) + Gathers console arguments, or prompts for them. + CreateUser(UUID.Zero, principalId, firstName, lastName, password, email, model); + public UserAccount CreateUser(UUID scopeID, UUID principalID, string firstName, string lastName, string password, string email, string model = "") + Looks almost identical to the OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs one above, but they add - + CreateDefaultAppearanceEntries(account.PrincipalID) + + + +account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email); + OpenSim/opensim-SC/OpenSim/Services/Interfaces/IUserAccountService.cs + public UserAccount(UUID scopeID, UUID principalID, string firstName, string lastName, string email) + Just holds the data in memory, in a dictionary I think. + OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs + public bool StoreUserAccount(UserAccount data) + Stuffs the data into a new UserAccountData() + m_Database.Store(d) + As far as I can tell, just dumps this data into the UserAccounts table - + FirstName, LastName, PrincipleID, ScopeID, Email, Created, UserLevel, UserFlags, UserTitle + PrincipleID is their randomly generated with no thought to collisions UUID. + ScopeID is 00000000-0000-0000-0000-000000000000 + Userlevel is 0 for most, -1 for Waki, determines if they can log on. Also higher for gods and things. + UserFlags, I think the only one is "64 god can login to this account using gods password. + UserTitle might default to "Local", or be configurable / and editable. + something something URL encoded "ServiceURLs" mumble + HomeURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GatekeeperURI= InventoryServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f AssetServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f ProfileServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f FriendsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f IMServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GroupsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f + Though most are either NULL, empty, or - + HomeURI= GatekeeperURI= InventoryServerURI= AssetServerURI= + Doesn't metion "active", which is always equal to 1 I guess. + + + +success = authenticationService.SetPassword(account.PrincipalID, password) + OpenSim/Services/AuthenticationService/AuthenticationServiceBase.cs + stores password details in "auth" table - + UUID + passwordSalt = Util.Md5Hash(UUID.Random().ToString()); + passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt); + accountType = "UserAccount"; + webLoginKey = UUID.Zero.ToString(); + + + +gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0)); + OpenSim/Services/UserAccountService/GridUserService.cs + Stores in database table GridUser + HomeRegionID, HomePosition, HomeLookAt + The other fields in that table - + UserID, LastRegionID, LastPosition, LastLookAt, Online (true or false), Login (timestamp or 0), Logout (timestamp or 0). + + + +success = inventoryService.CreateUserInventory(account.PrincipalID); + OpenSim/Services/InventoryService/XInventoryService.cs + Create a bunch of folders in the users inventory, of specific types. + rootFolder = ConvertToOpenSim(CreateFolder(principalID, UUID.Zero, (int)FolderType.Root, InventoryFolderBase.ROOT_FOLDER_NAME)); + XInventoryFolder[] sysFolders = GetSystemFolders(principalID, rootFolder.ID) + if (!Array.Exists(sysFolders, delegate(XInventoryFolder f) { if (f.type == (int)FolderType.Animation) return true; return false; })) + CreateFolder(principalID, rootFolder.ID, (int)FolderType.Animation, "Animations"); + FolderType.BodyPart, "Body Parts" + XInventoryFolder folder = CreateFolder(principalID, rootFolder.ID, (int)FolderType.CallingCard, "Calling Cards"); + folder = CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "Friends") + CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "All"); + FolderType.Clothing, "Clothing" + FolderType.CurrentOutfit, "Current Outfit" + FolderType.Favorites, "Favorites" + FolderType.Gesture, "Gestures") + FolderType.Landmark, "Landmarks" + FolderType.LostAndFound, "Lost And Found" + FolderType.Notecard, "Notecards" + FolderType.Object, "Objects" + FolderType.Snapshot, "Photo Album" + FolderType.LSLText, "Scripts" + FolderType.Sound, "Sounds" + FolderType.Texture, "Textures" + FolderType.Trash, "Trash" + + Stores in database inventoryFolders ???? + folderName, type, version = 1, folderID = UUID.Random(), agentID = principalID, parentFolderID = parentID + + + + +CreateDefaultAppearanceEntries(account.PrincipalID) + OpenSim/Services/UserAccountService/UserAccountService.cs + protected void CreateDefaultAppearanceEntries(UUID principalID) + Creates a bunch of "Default *" body parts and clothes, Ruth 1.0, links them in Inventories current outfit folder. + Creates a AvatarWearable[] and puts them all in it. + AvatarAppearance ap = new AvatarAppearance(); + ap.SetWearable(i, wearables[i]); + m_AvatarService.SetAppearance(principalID, ap); + + + + + +UserAccounts table - + UserFlags 64 is "allow gods to log in as me" + 0xf00 is membershipType, unles there's a title. Only sent to viewers I think. + 32 is Minors for estate banning purposes. + 4 is Anonymous for estate banning purposes. + 1 is AllowPublish in profile, but userprofile has this as separate field. + 2 is MaturePublish in profile, but userprofile has this as separate field. +Presence table - + UserID varchar(255) + RegionID char(36) + SessionID char(36) + SecureSessionID char(36) + LastSeen timestamp +tokens table (I think this is actually used for something) - + UUID char(36) + token varchar(255) current example looks like a UUID. + validity datetime +userdata (empty, can't find any actual usage in the source code, part of profiles) - + UserId char(36) primary index + TagId varchar(64) primary index + DataKey varchar(255) + DataVal varchar(255) +auth.webLoginKey seems to be some sort of passwordy type thing, though perhaps not actually hashed, rarely used, none of IG members have one. + + +PLAN- +. username +. password +. create login + +.check if it's a proper two word name +.login -> check if it's an existing account, get their UUID. + create toke_n_munchie + write session record + +create -> new user + create new UUID + check if it's an existing UUID + dbCount(, "UserAccounts", "PrincipleID='new-UUID'") + loop until we get a new one + create toke_n_munchie + write session record + + + Create -> + (wait a few seconds before showing this page) +. email +. email again +. password again +. DoB +. accept terms of service +. claim to be an adult +. confirm / cancel + + New user + UserAccounts.FirstName = ??? + UserAccounts.LastName = ??? + UserAccounts.Email = ??? + UserAccounts.Created = timestamp + UserAccounts.PrincipleID = randomly generate UUID, but check for collisions with other accounts. + It's a UNIQUE KEY. + UserAccounts.ScopeID = 00000000-0000-0000-0000-000000000000 + UserAccounts.Userlevel = -200 + UserAccounts.UserFlags = 64 + UserAccounts.UserTitle = newbie + UserAccounts.ServiceURLs = "" + UserAccounts.active = 0 + + auth.UUID = UserAccounts.PrincipleID + It's a PRIMARY KEY. + auth.passwordSalt = Util.Md5Hash(UUID.Random().ToString()) + auth.passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt) + auth.accountType = "UserAccount" + auth.webLoginKey (varchar(255)) = "00000000-0000-0000-0000-000000000000" + + userdata.UserId = UserAccounts.PrincipleID + userdata.TagId = "account creation data" + It's a UNIQUE KEY + userdata.DataKey = "DoB" + userdata.DataVal = ??? + + userdata.UserId = UserAccounts.PrincipleID + userdata.TagId = "account creation data" + userdata.DataKey = "timezone" + userdata.DataVal = ??? + + userdata.UserId = UserAccounts.PrincipleID + userdata.TagId = "account creation data" + userdata.DataKey = "Terms of service" + userdata.DataVal = "True" + + userdata.UserId = UserAccounts.PrincipleID + userdata.TagId = "account creation data" + userdata.DataKey = "claims to be an adult" + userdata.DataVal = "True" + + + Validated via email + (wait a few seconds before showing this page) + UserAccounts.Userlevel = -100 + UserAccounts.UserTitle = validated + + + Vouched for + userdata.UserId = UserAccounts.PrincipleID + userdata.TagId = "vouches" + userdata.DataKey = UUID of voucher + userdata.DataVal = timestamp of vouching + + UserAccounts.Userlevel = -50 + UserAccounts.UserTitle = vouched for + + + Admin approved + GridUser.UserID = UserAccounts.PrincipleID + It's a PRIMARY KEY. + GridUser.HomeRegionID = ??? + GridUser.HomePosition = ??? + GridUser.HomeLookAt = ??? + GridUser.LastRegionID = ??? + GridUser.LastPosition = ??? + GridUser.LastLookAt = ??? + GridUser.Online = False + GridUser.Login = 0 + GridUser.Logout = 0 + + UserAccounts.active = 1 + UserAccounts.Userlevel = 1 + UserAccounts.UserTitle = Member / Local / whatever + + Load the default IAR. + + +------------------------------------------------------------------- + +https://project-awesome.org/aleksandar-todorovic/awesome-c + A curated list of C good stuff. + +https://wolkykim.github.io/qdecoder/ + CGI library made by the qlibc guy, does support FCGI. + Might be a wrapper around the fcgi_stdio stuff I'm already using? + + +https://danielmiessler.com/study/http/ + A Security-focused HTTP Primer + Nothing much new except to say this about the Referer header - + "should not be used to make security decisions as it is controlled by the client" + Though others tell us to do precisely that. lol + + +------------------------------------------------------------------- + +apt install libmariadbclient-dev libapache2-mod-fcgid libssl1.0-dev uuid-dev spawn-fcgi + +BTW, everything is some BSD licence variation, except MariaDB, which is +some combination of GPLv2, 2+, 3, and BSD variants. With FOSS License +Exception, which seems to mean "Oracles GPL isn't viral". Though my +license is "fuck you viral GPL, that's not your decision to make for my +fucking code" BSD. + +------------------------------------------------------------------- + +Install / update / upgrade. + +I could keep the version number around. + Include version numbers / branches of dependencies. + Update will grab any security updates for the installed version. + Upgrade will upgrade to a chosen later different version. + Downgrade will downgrade to a chosen earlier different version. + +Note that we are currently using the LuaJIT 2.1.0-beta3 branch of the +main Luajit repo. Everything else is on their master branches. + +Bootstrap - + bootstrap.sh or bootstrap.bat + + Build the LuaJIT that comes with our source. It "builds out-of-the + box on most systems" and has no dependencies, other than a C build system. + + Or download a prebuilt LuaJIT from somewhere. + + After toybox has been LuaJITized. + + Build the LuaJIT that comes with our source. It "builds out-of-the + box on most systems" and has no dependencies, other than a C build system. + + Similar should apply to toybox, though it's our LuaJITized version. + Will need a specific miniconfig for this that doesn't include sledjchisl. + + Or download a prebuilt toybox+LuaJIT from a SledjHamr package repo. + +Install - + install.lua + + Will need a pre flight check if the dependencies are installed. + It checks if the system is built and has source. + Build it all. + Do the usual copy stuff to a directory thing. + Run "sledjchisl -install" in that directory. + Which does the usual "check health of system and fix up problems" thing, then quits instead of keep running. + The health check should include making sure our database creds exist / work. + +Update / upgrade / downgrade + install.lua -update + install.lua -upgrade + install.lua -downgrade + + Check if we are a binary only, or a source install. + wget new binaries / git pull new source + Toybox has a wget in pending, otherwise it only has ftpget. + Git is standalone outside of the system, but if you are + running from source, you likely have standard build tools + like git. + + +Yeah I hate things that have their own packaging system, for needing to +step outside the operating systems packaging system, and adding to the too +long list of stuff I have to deal with manually, and now I are one. lol + + +------------------------------------------------------------------- + +Time for a restructure of the web page / field / database stuff. + +Will need to include a "what page is this" cookie, or maybe query ?mode=add + + + +old validate UUID + define the UUID based UserAccounts db static dbRequest, fill it if needed. + if create + try to find an unused UUID + fill Rd->stuff with UUID + if confirm + check it's length + otherwise + check it's length + look it up, bitch if not found + If we found it, put level into Rd->database + fill Rd->stuff with UUID + +old validateName + define the name based UserAccounts db static dbRequest, fill it if needed. + Do the Lua file lookup, fill a tnm hash. + Do the database lookup, fill rows. + if login + convert tnm to Rd->database, or dbPull(rows) + fill Rd->stuff with name, UUID, and level + if create + complain if we found a record + try to find an unused UUID + fill Rd->database with new data + fill Rd->stuff with name, UUID, and level + +old validatePassword + define the UUID based auth db static dbRequest, fill it if needed. + if login + do the database lookup, fill rows + check if the name validation found us a UUID, fail login if it didn't + do the pasword+salt hash and compare + fill Rd->stuff with passwordHash and passwordSalt + if create + fill Rd->stuff with paswordHash and passwordSalt + if confirm + check if password hashess are the same + + + + + +freeSesh(Rd, linky, wipe) + linky - Rd->shs or Rd->lnk + %s/sessions/%s.lua or %s/sessions/%s.linky + wipe - wipe or delete session + wiping means remove session stuff from Rd->stuff + Which happens on - session failing to write, redirecting login form, showing login form if not confirm, vegOut (session timeout, bitchSession) + +newSesh(Rd, linky) + linky - old Rd->shs or a new Rd->lnk + setToken_n_munchie(Rd, linky); Only caller of setToken_n_munchie(Rd, linky); + + +setToken_n_munchie(Rd, linky) + linky - Rd->shs or Rd->lnk + %s/sessions/%s.lua or %s/sessions/%s.linky + !linky - actually set the cookies. + if error writing session file - freeSesh(Rd, linky, TRUE); + + +//validateSesh() +sessionValidate() + bitchSession() for bad session things. + sets chillOut for validated session linky. + Rd->chillOut = TRUE; + freeSesh(Rd, linky, FALSE); + Rd->func = (pageBuildFunction) loginPage; + Rd->doit = "logout"; + sets vegOut if the session timed out. + +//validatePassword() + sets chillOut for validated password on create. + +bitchSession() called if there's anything wrong with the session trackers, if we can't load / run the users Lua file, + sets vegOut + +account_HTML() + sets chillOut for POST confirm + createUser(Rd); + newSesh(Rd, TRUE); + Rd->chillOut = TRUE; + sets chillOut for POST login + Rd->chillOut = TRUE; + + + POST with no errors will + form == accountLogin freeSesh(Rd, FALSE, TRUE) + doit == login chillOut = TRUE + vegOut freeSesh(Rd, FALSE, TRUE); + else chillOut freeSesh(Rd, FALSE, FALSE); newSesh(Rd, FALSE); + else no Rd->shs.leaf newSesh(Rd, FALSE); + redirect to GET + otherwise + form == accountLogin + doit == confirm freeSesh(Rd, FALSE, TRUE) + newSesh(Rd, FALSE) + else if errors reeSesh(Rd, FALSE, FALSE) newSesh(Rd, FALSE) + show page + + + +LOGGED IN means that the session stored on disk has a valid UUID. + When creating a new user, we create a new UUID firstish. + + +accountLoginWeb() / accountOut() + freeSesh(Rd, FALSE, TRUE) + newSesh(Rd, FALSE) + +accountView() + freeSesh(Rd, FALSE, FALSE) + newSesh(Rd, FALSE) + +accountAdd() + Note that this is in two parts, first they click "create" on login page, then "confirm" on the account creation page. + + + +Account creation + accountLoginWeb() + "create" -> + Show accountCreateWeb and await confirmation. + accountCreateWeb() + "confirm" -> accountAdd() + create UUID + create user + store user + wipe old session + store new session with UUID, user is logged in now + create linky + email linky + Show usual logged in page. + "cancel" -> + + +------------------------------------------------------------------- + + +Maybe - /opt/opensim_SC/var/cache/sessions/uuid-uuid-uuid-uuid.logged symlink to session. + +https://localhost/sledjchisl.fcgi/account.html?user=account_name +https://localhost/sledjchisl.fcgi/account.html/users/account_name + logged in user is in the sesion, but they can view / vouch / edit / delete any other user depending on their access level + + +For logged in user, at the top show their name as linky to their accountView http://localhost/sledjchisl.fcgi/account.html/users/account_name + That accountView offers edit / logout button, etc. + Display account stuff, but not edit it until they hit the edit button. + +When showing other users + accountView, with edit / delete buttons if logged in user is high enough level. + +------------------------------------------------------------------- +------------------------------------------------------------------- +------------------------------------------------------------------- + + +BUGS! +----- +Redo the santize(), though that needs extensive changes each time we read Rd->cookies, Rd->queries, and Rd->body + diff --git a/src/sledjchisl/NOTES.txt b/src/sledjchisl/NOTES.txt deleted file mode 100644 index e80b8d9..0000000 --- a/src/sledjchisl/NOTES.txt +++ /dev/null @@ -1,479 +0,0 @@ -I'm re-purposing this for SledjHamr https://sledjhamr.org/git/docs/index.html - -The general structure of SledjHamr is a bunch of servers talking to each -other via Internet (or just local) connections. One of them is a web -server for assets, world data, and inventory. - -Originally I didn't think using a web based world client was a good idea, -however it might be better to have one, for reasons. Now I need a web -management console that can do all the things the current tmux console -can, including OpenSim console and commands. Plus account management for -users. I can also use a web based Jabber / XMPP front end to chat, IM, -and group chatter, which would run in the normal viewers web browser. -This provides a doorway into putting SledjHamr stuff in existing viewers -without needing them to support it. So a web based viewer now makes more -sense, and also means we can get away with not needing a viewer at all. - -Toybox itself doesn't include a web server, and I don't think there is -one on the roadmap. So we have to use an external web server, which was -a design goal of SledjHamr in the first place, using existing mature -HTTP infrastructure, coz that's already solved problems for a bunch of -things that plague OS/SL to this day. Clear your cache! Pffft. - -So sledjchisl.c will be the "love world server", though initially it just -drives OpenSim_SC in tmux via tmux commands to send keys and read output. -Later it might run opensim_SC directly and use STDIN and STDOUT to do -everything. It'll also provide the text management front end that runs -in the left tmux panel of the first window, which is why it's based on -boxes in the first place. Later still it can take over opensim_SC -functions as I move them out of mono. - -We will need a text, web, and GUI version of this management front end. -Hmmm, maybe don't need a GUI version, GUI users can just run a terminal. - -After much research, FastCGI / FCGI seems to be the most portable way of -interfacing with existing web servers. FCGI protocol closes STDERR and -STDOUT, and uses STDIN as it's two way communications channel to the web -server, so our FCGI module can't be used as the text management front -end. This is probably a good idea to keep them seperate anyway, for -security, coz the web server is exposed to the world, the console isn't. - -Currently sledjchisl.c tests to see if it's running in tmux already, if -it isn't it starts up tmux runs itself into this new tmux, then exits. -So it could also test if it's running from FCGI, and switch to web mode, -then it'll need to find the tmuxed instance to send commands to it. -Either via nails connection, or sending tmux commands via shell. - -FCGI has methods of dealing with auth and templates. B-) - -So for now I think I'll have the text and web management front ends in -sledjchisl.c, and the love world server as well. I can split them up -later if I need to. - - - - -I has Apache 2.4.25-3+deb9u9 - MariaDB 10.1.44-MariaDB - - -https://gist.github.com/dermesser/e2f9b66457ae19ebd116 - Multithreaded example in C. - - -------------------------------------------------------------------- - -Apache doesn't seem to support FCGI filter role, so I might have to do -without. Might be better anyway. - - -"A Filter is similar in functionality to a Responder that takes a data -file as a parameter. The difference is that with a Filter, both the data -file and the Filter itself can be access controlled using the Web -server's access control mechanisms, while a Responder that takes the name -of a data file as a parameter must perform its own access control checks -on the data file." - - Which is fine, our access control checks will be "Is this database - defined user already logged on via our FCGI script?". We should have - total control over that. I was planning on using the FCGI auth - mechanism anyway. - - -RESPONDER - web server sends FCGI_PARAMS - CONTENT_LENGTH - web server sends input body FCGI_STDIN - fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR - it has to finish reading FCGI_PARAMS first - fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE) - - -FILTER - filtered file has last modified time - web server sets FCGI_DATA_LAST_MOD accordingly - web server sends FCGI_PARAMS - CONTENT_LENGTH FCGI_DATA_LAST_MOD FCGI_DATA_LENGTH - web server sends input body FCGI_STDIN - web servers sends file over FCGI_DATA - fcgi app can ignore FCGI_DATA and use it's own cached copy based on FCGI_DATA_LAST_MOD - fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR - it has to finish reading FCGI_STDIN first, but not FCGI_DATA - fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE) - - -Soooo, FILTER might be slower anyway if we are caching the filtered file, -or mmapping it, coz filter has to start sending the filtered file, even -if it's to be served from cache. Plus no need to wait for FCGI_STDIN -before spewing it out. - - -Last update time for parameters, plus an update frequency. Once a minute. - - NOTE - SSI is a bit more complex than what I'm currently using. - https://en.wikipedia.org/wiki/Server_Side_Includes - - - - -. - - - - - - - - - - - - - - - https://www.w3.org/Jigsaw/Doc/User/SSI.html - Adds lots of others, including Java stuff. - Mine - - - - -------------------------------------------------------------------- - -Account creation process in the database. - -Apart from the usual input validation of things... - - -OpenSim/Server/Handlers/UserAccounts/UserAccountServerPostHandler.cs - byte[] CreateUser(Dictionary request) - Looks like their built in web front end, perhaps what is triggered by the console? - createdUserAccount - = ((UserAccountService)m_UserAccountService).CreateUser(scopeID, principalID, firstName, lastName, password, email, model); - -OpenSim/opensim-SC/OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs - An XML RPC interface to - - private UserAccount CreateUser(UUID scopeID, string firstName, string lastName, string password, string email) - account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email); - if (userAccountService.StoreUserAccount(account)) - success = authenticationService.SetPassword(account.PrincipalID, password) - gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0)); - success = inventoryService.CreateUserInventory(account.PrincipalID); - -OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs - Looks like the console command handler. - create user [ [ [ [ [ []]]]]] - Create a new user - protected void HandleCreateUser(string module, string[] cmdparams) - Gathers console arguments, or prompts for them. - CreateUser(UUID.Zero, principalId, firstName, lastName, password, email, model); - public UserAccount CreateUser(UUID scopeID, UUID principalID, string firstName, string lastName, string password, string email, string model = "") - Looks almost identical to the OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs one above, but they add - - CreateDefaultAppearanceEntries(account.PrincipalID) - - - -account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email); - OpenSim/opensim-SC/OpenSim/Services/Interfaces/IUserAccountService.cs - public UserAccount(UUID scopeID, UUID principalID, string firstName, string lastName, string email) - Just holds the data in memory, in a dictionary I think. - OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs - public bool StoreUserAccount(UserAccount data) - Stuffs the data into a new UserAccountData() - m_Database.Store(d) - As far as I can tell, just dumps this data into the UserAccounts table - - FirstName, LastName, PrincipleID, ScopeID, Email, Created, UserLevel, UserFlags, UserTitle - PrincipleID is their randomly generated with no thought to collisions UUID. - ScopeID is 00000000-0000-0000-0000-000000000000 - Userlevel is 0 for most, -1 for Waki, determines if they can log on. Also higher for gods and things. - UserFlags, I think the only one is "64 god can login to this account using gods password. - UserTitle might default to "Local", or be configurable / and editable. - something something URL encoded "ServiceURLs" mumble - HomeURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GatekeeperURI= InventoryServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f AssetServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f ProfileServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f FriendsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f IMServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GroupsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f - Though most are either NULL, empty, or - - HomeURI= GatekeeperURI= InventoryServerURI= AssetServerURI= - Doesn't metion "active", which is always equal to 1 I guess. - - - -success = authenticationService.SetPassword(account.PrincipalID, password) - OpenSim/Services/AuthenticationService/AuthenticationServiceBase.cs - stores password details in "auth" table - - UUID - passwordSalt = Util.Md5Hash(UUID.Random().ToString()); - passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt); - accountType = "UserAccount"; - webLoginKey = UUID.Zero.ToString(); - - - -gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0)); - OpenSim/Services/UserAccountService/GridUserService.cs - Stores in database table GridUser - HomeRegionID, HomePosition, HomeLookAt - The other fields in that table - - UserID, LastRegionID, LastPosition, LastLookAt, Online (true or false), Login (timestamp or 0), Logout (timestamp or 0). - - - -success = inventoryService.CreateUserInventory(account.PrincipalID); - OpenSim/Services/InventoryService/XInventoryService.cs - Create a bunch of folders in the users inventory, of specific types. - rootFolder = ConvertToOpenSim(CreateFolder(principalID, UUID.Zero, (int)FolderType.Root, InventoryFolderBase.ROOT_FOLDER_NAME)); - XInventoryFolder[] sysFolders = GetSystemFolders(principalID, rootFolder.ID) - if (!Array.Exists(sysFolders, delegate(XInventoryFolder f) { if (f.type == (int)FolderType.Animation) return true; return false; })) - CreateFolder(principalID, rootFolder.ID, (int)FolderType.Animation, "Animations"); - FolderType.BodyPart, "Body Parts" - XInventoryFolder folder = CreateFolder(principalID, rootFolder.ID, (int)FolderType.CallingCard, "Calling Cards"); - folder = CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "Friends") - CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "All"); - FolderType.Clothing, "Clothing" - FolderType.CurrentOutfit, "Current Outfit" - FolderType.Favorites, "Favorites" - FolderType.Gesture, "Gestures") - FolderType.Landmark, "Landmarks" - FolderType.LostAndFound, "Lost And Found" - FolderType.Notecard, "Notecards" - FolderType.Object, "Objects" - FolderType.Snapshot, "Photo Album" - FolderType.LSLText, "Scripts" - FolderType.Sound, "Sounds" - FolderType.Texture, "Textures" - FolderType.Trash, "Trash" - - Stores in database inventoryFolders ???? - folderName, type, version = 1, folderID = UUID.Random(), agentID = principalID, parentFolderID = parentID - - - - -CreateDefaultAppearanceEntries(account.PrincipalID) - OpenSim/Services/UserAccountService/UserAccountService.cs - protected void CreateDefaultAppearanceEntries(UUID principalID) - Creates a bunch of "Default *" body parts and clothes, Ruth 1.0, links them in Inventories current outfit folder. - Creates a AvatarWearable[] and puts them all in it. - AvatarAppearance ap = new AvatarAppearance(); - ap.SetWearable(i, wearables[i]); - m_AvatarService.SetAppearance(principalID, ap); - - - - - -UserAccounts table - - UserFlags 64 is "allow gods to log in as me" - 0xf00 is membershipType, unles there's a title. Only sent to viewers I think. - 32 is Minors for estate banning purposes. - 4 is Anonymous for estate banning purposes. - 1 is AllowPublish in profile, but userprofile has this as separate field. - 2 is MaturePublish in profile, but userprofile has this as separate field. -Presence table - - UserID varchar(255) - RegionID char(36) - SessionID char(36) - SecureSessionID char(36) - LastSeen timestamp -tokens table (I think this is actually used for something) - - UUID char(36) - token varchar(255) current example looks like a UUID. - validity datetime -userdata (empty, can't find any actual usage in the source code, part of profiles) - - UserId char(36) primary index - TagId varchar(64) primary index - DataKey varchar(255) - DataVal varchar(255) -auth.webLoginKey seems to be some sort of passwordy type thing, though perhaps not actually hashed, rarely used, none of IG members have one. - - -PLAN- -. username -. password -. create login - -.check if it's a proper two word name -.login -> check if it's an existing account, get their UUID. - create toke_n_munchie - write session record - -create -> new user - create new UUID - check if it's an existing UUID - dbCount(, "UserAccounts", "PrincipleID='new-UUID'") - loop until we get a new one - create toke_n_munchie - write session record - - - Create -> - (wait a few seconds before showing this page) -. email -. email again -. password again -. DoB -. accept terms of service -. claim to be an adult -. confirm / cancel - - New user - UserAccounts.FirstName = ??? - UserAccounts.LastName = ??? - UserAccounts.Email = ??? - UserAccounts.Created = timestamp - UserAccounts.PrincipleID = randomly generate UUID, but check for collisions with other accounts. - It's a UNIQUE KEY. - UserAccounts.ScopeID = 00000000-0000-0000-0000-000000000000 - UserAccounts.Userlevel = -200 - UserAccounts.UserFlags = 64 - UserAccounts.UserTitle = newbie - UserAccounts.ServiceURLs = "" - UserAccounts.active = 0 - - auth.UUID = UserAccounts.PrincipleID - It's a PRIMARY KEY. - auth.passwordSalt = Util.Md5Hash(UUID.Random().ToString()) - auth.passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt) - auth.accountType = "UserAccount" - auth.webLoginKey (varchar(255)) = "00000000-0000-0000-0000-000000000000" - - userdata.UserId = UserAccounts.PrincipleID - userdata.TagId = "account creation data" - It's a UNIQUE KEY - userdata.DataKey = "DoB" - userdata.DataVal = ??? - - userdata.UserId = UserAccounts.PrincipleID - userdata.TagId = "account creation data" - userdata.DataKey = "timezone" - userdata.DataVal = ??? - - userdata.UserId = UserAccounts.PrincipleID - userdata.TagId = "account creation data" - userdata.DataKey = "Terms of service" - userdata.DataVal = "True" - - userdata.UserId = UserAccounts.PrincipleID - userdata.TagId = "account creation data" - userdata.DataKey = "claims to be an adult" - userdata.DataVal = "True" - - - Validated via email - (wait a few seconds before showing this page) - UserAccounts.Userlevel = -100 - UserAccounts.UserTitle = validated - - - Vouched for - userdata.UserId = UserAccounts.PrincipleID - userdata.TagId = "vouches" - userdata.DataKey = UUID of voucher - userdata.DataVal = timestamp of vouching - - UserAccounts.Userlevel = -50 - UserAccounts.UserTitle = vouched for - - - Admin approved - GridUser.UserID = UserAccounts.PrincipleID - It's a PRIMARY KEY. - GridUser.HomeRegionID = ??? - GridUser.HomePosition = ??? - GridUser.HomeLookAt = ??? - GridUser.LastRegionID = ??? - GridUser.LastPosition = ??? - GridUser.LastLookAt = ??? - GridUser.Online = False - GridUser.Login = 0 - GridUser.Logout = 0 - - UserAccounts.active = 1 - UserAccounts.Userlevel = 1 - UserAccounts.UserTitle = Member / Local / whatever - - Load the default IAR. - - -------------------------------------------------------------------- - -https://project-awesome.org/aleksandar-todorovic/awesome-c - A curated list of C good stuff. - -https://wolkykim.github.io/qdecoder/ - CGI library made by the qlibc guy, does support FCGI. - Might be a wrapper around the fcgi_stdio stuff I'm already using? - - -https://danielmiessler.com/study/http/ - A Security-focused HTTP Primer - Nothing much new except to say this about the Referer header - - "should not be used to make security decisions as it is controlled by the client" - Though others tell us to do precisely that. lol - - -------------------------------------------------------------------- - -apt install libmariadbclient-dev libapache2-mod-fcgid libssl1.0-dev uuid-dev spawn-fcgi - -BTW, everything is some BSD licence variation, except MariaDB, which is -some combination of GPLv2, 2+, 3, and BSD variants. With FOSS License -Exception, which seems to mean "Oracles GPL isn't viral". Though my -license is "fuck you viral GPL, that's not your decision to make for my -fucking code" BSD. - -------------------------------------------------------------------- - -Install / update / upgrade. - -I could keep the version number around. - Include version numbers / branches of dependencies. - Update will grab any security updates for the installed version. - Upgrade will upgrade to a chosen later different version. - Downgrade will downgrade to a chosen earlier different version. - -Note that we are currently using the LuaJIT 2.1.0-beta3 branch of the -main Luajit repo. Everything else is on their master branches. - -Bootstrap - - bootstrap.sh or bootstrap.bat - - Build the LuaJIT that comes with our source. It "builds out-of-the - box on most systems" and has no dependencies, other than a C build system. - - Or download a prebuilt LuaJIT from somewhere. - - After toybox has been LuaJITized. - - Build the LuaJIT that comes with our source. It "builds out-of-the - box on most systems" and has no dependencies, other than a C build system. - - Similar should apply to toybox, though it's our LuaJITized version. - Will need a specific miniconfig for this that doesn't include sledjchisl. - - Or download a prebuilt toybox+LuaJIT from a SledjHamr package repo. - -Install - - install.lua - - Will need a pre flight check if the dependencies are installed. - It checks if the system is built and has source. - Build it all. - Do the usual copy stuff to a directory thing. - Run "sledjchisl -install" in that directory. - Which does the usual "check health of system and fix up problems" thing, then quits instead of keep running. - The health check should include making sure our database creds exist / work. - -Update / upgrade / downgrade - install.lua -update - install.lua -upgrade - install.lua -downgrade - - Check if we are a binary only, or a source install. - wget new binaries / git pull new source - Toybox has a wget in pending, otherwise it only has ftpget. - Git is standalone outside of the system, but if you are - running from source, you likely have standard build tools - like git. - - -Yeah I hate things that have their own packaging system, for needing to -step outside the operating systems packaging system, and adding to the too -long list of stuff I have to deal with manually, and now I are one. lol -- cgit v1.1