aboutsummaryrefslogtreecommitdiffstatshomepage
path: root/src/sledjchisl/NOTES.txt
blob: 9dde0164d458440798d50b89ab608a55d67bd77f (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
I'm re-purposing this for SledjHamr https://sledjhamr.org/git/docs/index.html

The general structure of SledjHamr is a bunch of servers talking to each
other via Internet (or just local) connections.  One of them is a web
server for assets, world data, and inventory. 

Originally I didn't think using a web based world client was a good idea,
however it might be better to have one, for reasons.  Now I need a web
management console that can do all the things the current tmux console
can, including OpenSim console and commands.  Plus account management for
users.  I can also use a web based Jabber / XMPP front end to chat, IM,
and group chatter, which would run in the normal viewers web browser. 
This provides a doorway into putting SledjHamr stuff in existing viewers
without needing them to support it.  So a web based viewer now makes more
sense, and also means we can get away with not needing a viewer at all.

Toybox itself doesn't include a web server, and I don't think there is
one on the roadmap.  So we have to use an external web server, which was
a design goal of SledjHamr in the first place, using existing mature
HTTP infrastructure, coz that's already solved problems for a bunch of
things that plague OS/SL to this day.  Clear your cache!  Pffft.

So sledjchisl.c will be the "love world server", though initially it just
drives OpenSim_SC in tmux via tmux commands to send keys and read output. 
Later it might run opensim_SC directly and use STDIN and STDOUT to do
everything.  It'll also provide the text management front end that runs
in the left tmux panel of the first window, which is why it's based on
boxes in the first place.  Later still it can take over opensim_SC
functions as I move them out of mono.

We will need a text, web, and GUI version of this management front end. 
Hmmm, maybe don't need a GUI version, GUI users can just run a terminal.

After much research, FastCGI / FCGI seems to be the most portable way of
interfacing with existing web servers.  FCGI protocol closes STDERR and
STDOUT, and uses STDIN as it's two way communications channel to the web
server, so our FCGI module can't be used as the text management front
end.  This is probably a good idea to keep them seperate anyway, for
security, coz the web server is exposed to the world, the console isn't.

Currently sledjchisl.c tests to see if it's running in tmux already, if
it isn't it starts up tmux runs itself into this new tmux, then exits. 
So it could also test if it's running from FCGI, and switch to web mode,
then it'll need to find the tmuxed instance to send commands to it. 
Either via nails connection, or sending tmux commands via shell.

FCGI has methods of dealing with auth and templates.  B-)

So for now I think I'll have the text and web management front ends in
sledjchisl.c, and the love world server as well.  I can split them up
later if I need to.




I has	Apache 2.4.25-3+deb9u9
	MariaDB 10.1.44-MariaDB


https://gist.github.com/dermesser/e2f9b66457ae19ebd116
    Multithreaded example in C.


-------------------------------------------------------------------

Apache doesn't seem to support FCGI filter role, so I might have to do
without.  Might be better anyway.


"A Filter is similar in functionality to a Responder that takes a data
file as a parameter. The difference is that with a Filter, both the data
file and the Filter itself can be access controlled using the Web
server's access control mechanisms, while a Responder that takes the name
of a data file as a parameter must perform its own access control checks
on the data file."

    Which is fine, our access control checks will be "Is this database
    defined user already logged on via our FCGI script?".  We should have
    total control over that.  I was planning on using the FCGI auth
    mechanism anyway.


RESPONDER
    web server sends FCGI_PARAMS
	CONTENT_LENGTH
    web server sends input body FCGI_STDIN
    fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR
	it has to finish reading FCGI_PARAMS first
    fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE)


FILTER
    filtered file has last modified time
	web server sets FCGI_DATA_LAST_MOD accordingly
    web server sends FCGI_PARAMS
	CONTENT_LENGTH   FCGI_DATA_LAST_MOD   FCGI_DATA_LENGTH
    web server sends input body FCGI_STDIN
    web servers sends file over FCGI_DATA
    fcgi app can ignore FCGI_DATA and use it's own cached copy based on FCGI_DATA_LAST_MOD
    fcgi app sends result data over FCGI_STDOUT and error messages over FCGI_STDERR
	it has to finish reading FCGI_STDIN first, but not FCGI_DATA
    fcgi app sends FCGI_END_REQUEST(protocolStatus = FCGI_REQUEST_COMPLETE)


Soooo, FILTER might be slower anyway if we are caching the filtered file,
or mmapping it, coz filter has to start sending the filtered file, even
if it's to be served from cache.  Plus no need to wait for FCGI_STDIN
before spewing it out.


Last update time for parameters, plus an update frequency.  Once a minute.

    NOTE - SSI is a bit more complex than what I'm currently using.
	https://en.wikipedia.org/wiki/Server_Side_Includes
	    <!--#include virtual="menu.cgi" -->
	    <!--#include file="footer.html" -->
	    <!--#exec cgi="/cgi-bin/foo.cgi" -->
	    <!--#exec cmd="ls -l" -->
.	    <!--#echo var="REMOTE_ADDR" -->
	    <!--#config timefmt="%y %m %d" -->
	    <!--#config sizefmt="bytes" -->
	    <!--#config errmsg="SSI command failed!" -->
	    <!--#flastmod virtual="index.html" -->
	    <!--#fsize file="script.pl" -->
	    <!--#if expr="${Sec_Nav}" -->
	    <!--#include virtual="secondary_nav.txt" -->
	    <!--#elif expr="${Pri_Nav}" -->
	    <!--#include virtual="primary_nav.txt" -->
	    <!--#else -->
	    <!--#include virtual="article.txt" -->
	    <!--#endif -->
	    <!--#set var="foo" value="bar" -->
	    <!--#printenv -->
	https://www.w3.org/Jigsaw/Doc/User/SSI.html
	    Adds lots of others, including Java stuff.
	Mine
	    <!--#lua lua="print(table[key])" -->
	    <!--#lua file="/path/to/script.lua" -->
	    <!--#lua virtual="https://example.com/script.lua" -->

-------------------------------------------------------------------

Account creation process in the database.

Apart from the usual input validation of things...


OpenSim/Server/Handlers/UserAccounts/UserAccountServerPostHandler.cs
    byte[] CreateUser(Dictionary<string, object> request)
	Looks like their built in web front end, perhaps what is triggered by the console?
        createdUserAccount
            = ((UserAccountService)m_UserAccountService).CreateUser(scopeID, principalID, firstName, lastName, password, email, model);

OpenSim/opensim-SC/OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs
    An XML RPC interface to -
	private UserAccount CreateUser(UUID scopeID, string firstName, string lastName, string password, string email)
	    account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email);
	    if (userAccountService.StoreUserAccount(account))
	    success = authenticationService.SetPassword(account.PrincipalID, password)
	    gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0));
	    success = inventoryService.CreateUserInventory(account.PrincipalID);

OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs
    Looks like the console command handler.
	create user [<first> [<last> [<pass> [<email> [<user id> [<model>]]]]]] - Create a new user
	    protected void HandleCreateUser(string module, string[] cmdparams)
		Gathers console arguments, or prompts for them.
		CreateUser(UUID.Zero, principalId, firstName, lastName, password, email, model);
	    public UserAccount CreateUser(UUID scopeID, UUID principalID, string firstName, string lastName, string password, string email, string model = "")
		Looks almost identical to the OpenSim/ApplicationPlugins/RemoteController/RemoteAdminPlugin.cs one above, but they add -
		    CreateDefaultAppearanceEntries(account.PrincipalID)



account = new UserAccount(scopeID, UUID.Random(), firstName, lastName, email);
    OpenSim/opensim-SC/OpenSim/Services/Interfaces/IUserAccountService.cs
	public UserAccount(UUID scopeID, UUID principalID, string firstName, string lastName, string email)
	    Just holds the data in memory, in a dictionary I think.
    OpenSim/opensim-SC/OpenSim/Services/UserAccountService/UserAccountService.cs
	public bool StoreUserAccount(UserAccount data)
	    Stuffs the data into a new UserAccountData()
	    m_Database.Store(d)
		As far as I can tell, just dumps this data into the UserAccounts table -
		    FirstName, LastName, PrincipleID, ScopeID, Email, Created, UserLevel, UserFlags, UserTitle
			PrincipleID is their randomly generated with no thought to collisions UUID.
			ScopeID is 00000000-0000-0000-0000-000000000000
			Userlevel is 0 for most, -1 for Waki, determines if they can log on.  Also higher for gods and things.
			UserFlags, I think the only one is "64 god can login to this account using gods password.
			UserTitle might default to "Local", or be configurable / and editable.
		    something something URL encoded "ServiceURLs" mumble
			HomeURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GatekeeperURI= InventoryServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f AssetServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f ProfileServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f FriendsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f IMServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f GroupsServerURI=http%3a%2f%2fgrid.infinitegrid.org%3a8002%2f
			Though most are either NULL, empty, or -
			    HomeURI= GatekeeperURI= InventoryServerURI= AssetServerURI=
		    Doesn't metion "active", which is always equal to 1 I guess.



success = authenticationService.SetPassword(account.PrincipalID, password)
    OpenSim/Services/AuthenticationService/AuthenticationServiceBase.cs
	stores password details in "auth" table -
	    UUID
	    passwordSalt = Util.Md5Hash(UUID.Random().ToString());
	    passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt);
	    accountType = "UserAccount";
	    webLoginKey = UUID.Zero.ToString();



gridUserService.SetHome(account.PrincipalID.ToString(), home.RegionID, new Vector3(128, 128, 0), new Vector3(0, 1, 0));
    OpenSim/Services/UserAccountService/GridUserService.cs
	Stores in database table GridUser
	    HomeRegionID, HomePosition, HomeLookAt
	The other fields in that table -
	    UserID, LastRegionID, LastPosition, LastLookAt, Online (true or false), Login (timestamp or 0), Logout (timestamp or 0).



success = inventoryService.CreateUserInventory(account.PrincipalID);
    OpenSim/Services/InventoryService/XInventoryService.cs
	Create a bunch of folders in the users inventory, of specific types.
	    rootFolder = ConvertToOpenSim(CreateFolder(principalID, UUID.Zero, (int)FolderType.Root, InventoryFolderBase.ROOT_FOLDER_NAME));
	    XInventoryFolder[] sysFolders = GetSystemFolders(principalID, rootFolder.ID)
	    if (!Array.Exists(sysFolders, delegate(XInventoryFolder f) { if (f.type == (int)FolderType.Animation) return true; return false; }))
		CreateFolder(principalID, rootFolder.ID, (int)FolderType.Animation, "Animations");
		    FolderType.BodyPart, "Body Parts"
		XInventoryFolder folder = CreateFolder(principalID, rootFolder.ID, (int)FolderType.CallingCard, "Calling Cards");
		folder = CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "Friends")
		CreateFolder(principalID, folder.folderID, (int)FolderType.CallingCard, "All");
		    FolderType.Clothing, "Clothing"
		    FolderType.CurrentOutfit, "Current Outfit"
		    FolderType.Favorites, "Favorites"
		    FolderType.Gesture, "Gestures")
		    FolderType.Landmark, "Landmarks"
		    FolderType.LostAndFound, "Lost And Found"
		    FolderType.Notecard, "Notecards"
		    FolderType.Object, "Objects"
		    FolderType.Snapshot, "Photo Album"
		    FolderType.LSLText, "Scripts"
		    FolderType.Sound, "Sounds"
		    FolderType.Texture, "Textures"
		    FolderType.Trash, "Trash"

		    Stores in database inventoryFolders ????
			folderName, type, version = 1, folderID = UUID.Random(), agentID = principalID, parentFolderID = parentID




CreateDefaultAppearanceEntries(account.PrincipalID)
    OpenSim/Services/UserAccountService/UserAccountService.cs
	protected void CreateDefaultAppearanceEntries(UUID principalID)
	Creates a bunch of "Default *" body parts and clothes, Ruth 1.0, links them in Inventories current outfit folder.
	Creates a AvatarWearable[] and puts them all in it.
	AvatarAppearance ap = new AvatarAppearance();
	    ap.SetWearable(i, wearables[i]);
	m_AvatarService.SetAppearance(principalID, ap);
	    




UserAccounts table -
    UserFlags		64 is "allow gods to log in as me"
			0xf00 is membershipType, unles there's a title.  Only sent to viewers I think.
			32 is Minors   for estate banning purposes.
			4 is Anonymous for estate banning purposes.
			1 is AllowPublish  in profile, but userprofile has this as separate field.
			2 is MaturePublish in profile, but userprofile has this as separate field.
Presence table -
    UserID		varchar(255)
    RegionID		char(36)
    SessionID		char(36)
    SecureSessionID	char(36)
    LastSeen		timestamp
tokens table (I think this is actually used for something) -
    UUID		char(36)
    token		varchar(255)	current example looks like a UUID.
    validity		datetime
userdata (empty, can't find any actual usage in the source code, part of profiles) -
    UserId		char(36)	primary index
    TagId		varchar(64)	primary index
    DataKey		varchar(255)
    DataVal		varchar(255)
auth.webLoginKey seems to be some sort of passwordy type thing, though perhaps not actually hashed, rarely used, none of IG members have one.


PLAN-
.    username
.    password
.    create   login

.check if it's a proper two word name
.login -> check if it's an existing account, get their UUID.
    create toke_n_munchie
    write session record

create -> new user
    create new UUID
    check if it's an existing UUID
	dbCount(, "UserAccounts", "PrincipleID='new-UUID'")
    loop until we get a new one
    create toke_n_munchie
    write session record


    Create ->
    (wait a few seconds before showing this page)
.    email
.    email again
.    password again
.    DoB
.    accept terms of service
.    claim to be an adult
.    confirm / cancel

    New user
	UserAccounts.FirstName = ???
	UserAccounts.LastName = ???
	UserAccounts.Email = ???
	UserAccounts.Created = timestamp
	UserAccounts.PrincipleID = randomly generate UUID, but check for collisions with other accounts.
							    It's a UNIQUE KEY.
	UserAccounts.ScopeID = 00000000-0000-0000-0000-000000000000
	UserAccounts.Userlevel = -200
	UserAccounts.UserFlags = 64
	UserAccounts.UserTitle = newbie
	UserAccounts.ServiceURLs = ""
	UserAccounts.active = 0

	auth.UUID = UserAccounts.PrincipleID
							    It's a PRIMARY KEY.
	auth.passwordSalt = Util.Md5Hash(UUID.Random().ToString())
	auth.passwdHash = Util.Md5Hash(Util.Md5Hash(password) + ":" + passwordSalt)
	auth.accountType = "UserAccount"
	auth.webLoginKey (varchar(255)) = "00000000-0000-0000-0000-000000000000"

	userdata.UserId = UserAccounts.PrincipleID
	userdata.TagId = "account creation data"
							    It's a UNIQUE KEY
	userdata.DataKey = "DoB"
	userdata.DataVal = ???

	userdata.UserId = UserAccounts.PrincipleID
	userdata.TagId = "account creation data"
	userdata.DataKey = "timezone"
	userdata.DataVal = ???

	userdata.UserId = UserAccounts.PrincipleID
	userdata.TagId = "account creation data"
	userdata.DataKey = "Terms of service"
	userdata.DataVal = "True"

	userdata.UserId = UserAccounts.PrincipleID
	userdata.TagId = "account creation data"
	userdata.DataKey = "claims to be an adult"
	userdata.DataVal = "True"


    Validated via email
    (wait a few seconds before showing this page)
	UserAccounts.Userlevel = -100
	UserAccounts.UserTitle = validated


    Vouched for
	userdata.UserId = UserAccounts.PrincipleID
	userdata.TagId = "vouches"
	userdata.DataKey = UUID of voucher
	userdata.DataVal = timestamp of vouching

	UserAccounts.Userlevel = -50
	UserAccounts.UserTitle = vouched for


    Admin approved
	GridUser.UserID = UserAccounts.PrincipleID
							    It's a PRIMARY KEY.
	GridUser.HomeRegionID = ???
	GridUser.HomePosition = ???
	GridUser.HomeLookAt = ???
	GridUser.LastRegionID = ???
	GridUser.LastPosition = ???
	GridUser.LastLookAt = ???
	GridUser.Online = False
	GridUser.Login = 0
	GridUser.Logout = 0

	UserAccounts.active = 1
	UserAccounts.Userlevel = 1
	UserAccounts.UserTitle = Member / Local / whatever

	Load the default IAR.


-------------------------------------------------------------------

https://project-awesome.org/aleksandar-todorovic/awesome-c
    A curated list of C good stuff.

https://wolkykim.github.io/qdecoder/
    CGI library made by the qlibc guy, does support FCGI.
    Might be a wrapper around the fcgi_stdio stuff I'm already using?


https://danielmiessler.com/study/http/
    A Security-focused HTTP Primer
	Nothing much new except to say this about the Referer header -
	    "should not be used to make security decisions as it is controlled by the client"
	    Though others tell us to do precisely that.  lol


-------------------------------------------------------------------

apt install libmariadbclient-dev libapache2-mod-fcgid libssl1.0-dev uuid-dev spawn-fcgi

-------------------------------------------------------------------

Merge it into OpenSim-SC.

    Complication - I had already added the boxes + early sledjchisl.c to opensim-SC.
	I may have to revert that lot, only a few minor commits, which are already part of the main boxes.
	d9c772712e27c8e25fab0d17555a9bc11017a125
	d4ea3e50173df1ad646bdb7dc802f5d320b7e511
	10ed36b3452ce6373175112716b043047dc896a9
	2f66c46e7ce18d60cd1f565880fb7762b3399ccb
	34c5ee4c2a489a506e93d5b303fbc80b263747f0 is the commit that added it.
	f9bfa831d1ccaa973c42042584510e1a724ddaef the one before it.
	It's been pushed up to the sledjhamr.org repo.

	git revert d9c772712e27c8e25fab0d17555a9bc11017a125 d4ea3e50173df1ad646bdb7dc802f5d320b7e511 10ed36b3452ce6373175112716b043047dc896a9 2f66c46e7ce18d60cd1f565880fb7762b3399ccb 34c5ee4c2a489a506e93d5b303fbc80b263747f0
	git remote add -f boxes ~/MyLinux_3/TOYBOX/sledjchisl
	git merge --allow-unrelated-histories boxes/master
	git remote rm boxes


https://medium.com/altcampus/how-to-merge-two-or-multiple-git-repositories-into-one-9f8a5209913f
https://medium.com/@checko/merging-two-git-repositories-into-one-preserving-the-git-history-4e20d3fafa4e
https://thoughts.t37.net/merging-2-different-git-repositories-without-losing-your-history-de7a06bba804
https://blog.doismellburning.co.uk/merging-two-git-repositories/

Other much more complicated variations.

http://mbork.pl/2019-08-19_Transplanting_a_directory_to_another_Git_repository


-------------------------------------------------------------------

Install / update / upgrade.

I could keep the version number around.
    Include version numbers / branches of dependencies.
    Update will grab any security updates for the installed version.
    Upgrade   will upgrade   to a chosen later    different version.
    Downgrade will downgrade to a chosen earlier  different version.

Note that we are currently using the LuaJIT 2.1.0-beta3 branch of the
main Luajit repo.  Everything else is on their master branches.

Bootstrap -
    bootstrap.sh or bootstrap.bat

    Build the LuaJIT that comes with our source.  It "builds out-of-the
    box on most systems" and has no dependencies, other than a C build system.

    Or download a prebuilt LuaJIT from somewhere.

  After toybox has been LuaJITized.

    Build the LuaJIT that comes with our source.  It "builds out-of-the
    box on most systems" and has no dependencies, other than a C build system.

    Similar should apply to toybox, though it's our LuaJITized version.
	Will need a specific miniconfig for this that doesn't include sledjchisl.

    Or download a prebuilt toybox+LuaJIT from a SledjHamr package repo.

Install -
    install.lua

	Will need a pre flight check if the dependencies are installed.
	It checks if the system is built and has source.
	    Build it all.
	Do the usual copy stuff to a directory thing.
	Run "sledjchisl -install" in that directory.
	    Which does the usual "check health of system and fix up problems" thing, then quits instead of keep running.
		The health check should include making sure our database creds exist / work.

Update / upgrade / downgrade
    install.lua -update
    install.lua -upgrade
    install.lua -downgrade

    Check if we are a binary only, or a source install.
	wget new binaries / git pull new source
	    Toybox has a wget in pending, otherwise it only has ftpget.
	    Git is standalone outside of the system, but if you are
	    running from source, you likely have standard build tools
	    like git.


Yeah I hate things that have their own packaging system, for needing to
step outside the operating systems packaging system, and adding to the too
long list of stuff I have to deal with manually, and now I are one.  lol