aboutsummaryrefslogtreecommitdiffstatshomepage
diff options
context:
space:
mode:
authorDavid Walter Seikel2014-05-26 20:28:47 +1000
committerDavid Walter Seikel2014-05-26 20:28:47 +1000
commitbe5cf9409f34c478af64749db5cb5f652ecb268d (patch)
tree92a54446b9cd0f29558e9455bd3a5a2c2e496e90
parentTODO++ (diff)
downloadSledjHamr-be5cf9409f34c478af64749db5cb5f652ecb268d.zip
SledjHamr-be5cf9409f34c478af64749db5cb5f652ecb268d.tar.gz
SledjHamr-be5cf9409f34c478af64749db5cb5f652ecb268d.tar.bz2
SledjHamr-be5cf9409f34c478af64749db5cb5f652ecb268d.tar.xz
Lots of docs about design goals and methods, plus a few clean ups.
-rw-r--r--README408
1 files changed, 406 insertions, 2 deletions
diff --git a/README b/README
index fe91a89..f1ca938 100644
--- a/README
+++ b/README
@@ -16,13 +16,413 @@ It's all very experimental at the moment, and currently does nothing
16useful. 16useful.
17 17
18 18
19Design goals and methods.
20=========================
21
22As mentioned, this is all highly experimental. Everything you find here
23is subject to change without notice. We do have some overall design
24goals and methods though.
25
26Simulating a world is what virtual world software does. The real world
27is full of lots and lots of stuff, and lots of that stuff has lots and
28lots of fiddly little details to make it work. So simulating a world
29involves thousands, if not millions, of fiddly little details.
30
31So yes you can have a working model of a car that you can actually get
32inside and drive around a virtual world. It doesn't have to have an
33actually working internal combustion engine, GPS, radio, or air
34conditioning. The "engine" could just be a roughly engine shaped mesh
35object with engine like textures wrapped around it to make it look like
36a car engine. Or the "engine" could just be a red cube on the front of
37the car. The "air conditioning" could just be a photo of a grill that's
38part of the texture used for the "dash board". Same for the "radio",
39"GPS", or pretty much most of the other fiddly bits of a real car. In
40virtual worlds you can program a bare cube to act like a car that you
41sit on and drive, the rest is mostly for looks.
42
43There's no such thing as virtual wind in your hair, or smog in your
44nostrils, but if that eventualy gets invented, a generic virtual world
45system should be able to hook into it easily enough.
46
47Generic.
48--------
49
50The trick is to design generic stuff that can fake most if it well
51enough to at least not annoy people. Too much.
52
53My experience is that if you do a good job of designing small useful
54generic code, it becomes very easy to build things you never dreamed of
55when you made the small generic bits. It is also part of the Unix
56philosophy, and you can do almost anything with the usual collection of
57small Unix tools.
58
59So the design of SledjHamr involves lots of little bits of generic code
60that all work together seemlessly. A major design goal is to be as
61generic, yet useful, as possible, to support future stuff that hasn't
62been invented yet. As well as supporting stuff we need to do now.
63
64If we do this right, then future work becomes really really easy.
65Virtual world development can then accelerate.
66
67Small.
68------
69
70Small is the next major design goal. For far to long people have argued
71that computer storage, memory, and CPU power are all very cheap, so why
72bother making things small. Any long term computer user might have
73noticed that despite our modern computers being many orders of magnitude
74faster and bigger than they where a decade or two ago, everything runs
75slower than it did then. This proves the falacy of "everything is cheap
76and getting cheaper, lets be wasteful" theory. People are just too
77comfortable with their wasteful habits. Not to mention that the
78computer industry loves to get every one to throw out their computers
79every couple of years and replace them with new shiny ones, just to make
80big profits. So every year normal software needs more powerful
81computers, just to do what it did fine last year.
82
83Even a small virtual world can use up huge amounts of storage, large
84amounts of memory and CPU power, and large amounts of network bandwidth.
85This is the problem with simulating worlds, the bits might be small, but
86there's an aweful lot of them. You can run a busy web site on a $5 per
87month web host, but you need to spend at least $100 per month on a
88hosted server that's powerful enough to run a small OpenSim based
89virtual world.
90
91The problem is scaling up to the level of detail needed. So keeping
92things small is an important design goal. The generic design mentioned
93above works for keeping things small, since you need less special
94purpose code for all those special purpose fiddly details.
95
96See the "Fast" section below about keeping network protocol and file
97sizes small.
98
99Fast.
100-----
101
102Ask any current virtual world user, at least of the types based on SL
103technology, what the number one biggest problem is and they will tell
104you it's lag. So speed is important to everyone. Which makes me wonder
105why people use slow bloated things like interpretted scripting languages
106and human readable network protocols / file formats for damn near
107everything?
108
109People use interpretted scripting languages coz it's easy and convenient
110for them. Not so convenient for the user though when it causes more
111lag. People invent human readable network protocols and file formats
112coz it makes it easy for them to read when they need to debug the
113system. But 99.9999% of the time these human readable network and file
114formats are NEVER READ BY HUMANS. Computers don't find them easy to
115read at all. It takes more effort for the computer to figure it out,
116more space on your hard drive, more space in your memory, and more
117bandwidth to send. All they do is slow everything down massively most
118of the time.
119
120So basically a small handful of developers are taking the easy way out,
121but that inconveniences the vast majority of people. Mind you, you cant
122always blame the developers, money is involved, which means people are
123forced by management to take what ever shortcuts they can now, even if
124it means pain later. That's economics for you.
125
126We wont follow that bullshit here.
127
128Once again it's a matter of scale for virtual worlds. Each part by
129itself might be barely fast enough that people don't notice, but it all
130adds up when you deal with the huge multitude of fiddly little details
131in a virtual world. Which results in the number one problem being ...
132lag. Often everything slows down so much it becomens unusable.
133
134We can have our cake and eat it to. So long as the crucial heavily used
135parts of the system that need speed are written efficiently in a
136decently fast language, then we can still use easy interpretted
137scripting languages for other parts that wont suffer from scaling
138issues. So long as we stick with efficient binary based network
139protocols and file formats, the tiny percentage of developers that need
140to actually read it can use specialised tools for that job that turn
141them into human readable forms, but not slow down every body else.
142
143It's my experience, and that of other people that work with code that
144needs to be low level, fast, and small, that the C language, with help
145from assembler sometimes, is the best choice for fast working code.
146Assembler and C is used for OS kernels, embedded software, and other
147things that need to be efficient. So C is the major language used for
148SledjHamr. Bits of assembler might be used if needed.
149
150For the interpretted scripting language of choice used in SledjHamr, I
151chose Lua. It's very generic in nature with it's wonderful tables and
152meta tables. It's tiny, designed to be embedded inside other languages.
153With the LuaJIT just in time compiler, it's the fastest scripting
154language around. These three things fit our major design goals. On top
155of those things, it's popular with other 3D online environments,
156especially WoW. It's also very easy to use and can be quite fast for
157development, which is why you use a scripting language in the first
158place. The ability to be embedded in other code means that Lua tends to
159actually be embedded in many other projects. Which can then use Lua to
160build all manner of oddball mash ups. Exactly what virtual worlds need.
161
162While on the subject of languages, see the stuff below about EFL. Also,
163when Apple based their new version of Mac Os X on BSD Unix, the OS wars
164where over. Everything now is some sort of Unix, except Windows. Hell,
165iPhones OS is based on Mac OS X, and Android is based on Linux, so even
166the great majority of phones these days are Unix. Every one is a Unix
167user, wether they know it or not.
168
169Windows makes a nod to being Posix compliant, though it's barely a nod.
170Cygwin however can be installed on Windows to make it more like the Unix
171used in the rest of the world. So in general Posix and Unix style stuff
172should be used. Try to keep the Windows specific stuff well isolated,
173small, and preferably based on Cygwin. This should lead to A) maximum
174portability. B) Less stress for the SledjHamr head developer, who got
175burnt out working with Windows crap long ago and really really REALLY
176wants to avoid it as much as possible now. The OS wars are over, Unix
177won, fucking get over it Microsoft.
178
179Modular.
180--------
181
182This is actually part of being generic and small. If everything is a
183small module, then you only need to use those modules that do the things
184you are doing right now. The rest you don't need, so you don't load
185them up. No need to cater for all of those fiddly little details in one
186massive world simulating blob, break it down into small modular and
187generic bits that talk to each other using common protocols and
188standards.
189
190This can also be useful to support the great variety of other file
191formats and protocols already being used. Need to support some obscure
192file format no one has heard of? Write a small module for it. All the
193major file formats can be supported out of the box. Want to join up
194with an obscure virtual world platform? Write a module to support it.
195All the common virtual world tasks will be already supported by
196SledjHamr, you just need to translate between the file format or network
197protocol in your module. OpenSim, SL, and later Cobalt, will be the
198initially supported virtual world network protocols.
199
200For the purposes of using the old SL type technology as a crutch during
201development, being modular makes a lot of sense. When any given module
202is good enough to start using, we can hook it up to Imprudence or
203OpenSim to start using it for real. When that module gets mature, we
204can rip the relevant code out of those older systems. Eventually those
205older code bases morph into SledjHamr based ones.
206
207Standards based.
208----------------
209
210OK, we will cheat a bit here. No such thing as a decently small and
211fast virtual world protocol standard, so we are inventing our own.
212
213On the other hand, we intend to use open standards based network
214protocols (even if they are bloated human readable ones) to take care of
215the various bits not specifically related to virtual worlds. Like
216Jabber/XMMP for text communications, HTTP for asset transfers, etc.
217These protocols are well used, well understood, and in general not TOO
218bloated for their intended use cases. These use cases are close enough
219to our own that we can leverage this huge installed base of code and
220expertise to deal with the problems that virtual worlds would share.
221Much less work for us all. People can use the software they are used to
222to interface with those bits.
223
224Standard texture file formats tend to be very well compressed, and the
225libraries for them more or less mature. Lots of research and
226standardisation effort has gone into the major popular ones. So we
227don't really need to deal with that. Problem solved already. Well,
228except for JPEG 2000, which outside of SL and medical use, no one has
229even heard of. It's library support is not so good, and made even wosre
230by the fact that the best decoding algorithm for it is patented and out
231of reach to open source developers. Still, lots and lots of SL and
232OpenSim content is in JPEG 2000 format, and it is good as a network
233texture format, so we just have to deal with it.
234
235Mesh file formats are a bit less standardised, and a bit less well
236compressed, but the libg3d mesh loading library deals with that well
237enough, and it's good to be able to use all that content out there. On
238the other hand, OpenSim OAR and IAR files are some what common, but best
239to use just for importing OpenSim based worlds into SledjHamr. They are
240not pretty on the inside. On the gripping hand, SL viewers never really
241standardised on prim file formats. In general you need to use the
242viewer you used to export to import them. However, this is the sort of
243thing the modularity is made for. Write small modules to deal with
244these SL inspired file formats. Make them libg3d modules.
245
246OpenGL is the worlds open 3D standard, and even works on Windows. It's
247well known that Microsoft (at least in the past) made their OpenGL
248implementation slower than their own 3D system, specifically so they
249could claim theirs was better. OpenGL works fine on Windows, lets use
250it. Any slowness issues we just blame on Microsoft.
251
252Secure.
253-------
254
255One of our developers is a cypherpunk / cryptogeek / whatever term she's
256comfortable being labelled as. Our team is very privacy focused. So
257security and privacy are important goals as well. Small modular code is
258better for security, is there is less code to look at to do security
259audits, and less places for things to go wrong, or escape attention.
260Commonly used security and privacy protocols also tend to be built in
261modular ways, so we can make use of these things ourselves.
262
263Pretty and skinable.
264--------------------
265
266The Enlightenment Foundation Libraries (EFL) where written to support
267the Enlightenment X window manager, DR17 (E17). Enlightenment is known for
268it's eye candy. The early DR16 version was popular, but known to be a
269heavy resource user. That was many years ago, and the most popular X
270window managers these days far surpassed Enlightenment DR16 for bloat,
271without coming close for eye candy.
272
273E17 was rewritten from scratch, and I helped with that rewrite in
274various ways. E17 is a lot less "bloated", but I think that's mainly
275coz everything else passed it in bloatedness, even though much work was
276done to ensure E17 would work great on small embedded systems. I have
277used EFL on professional embedded systems, and it worked well, even on
278the tiny x486 embedded board I used. In general EFL fits well with my
279personal coding standards, and fits well with the SledjHamr goals. This
280is why I continue to work with it, and used it for SledjHamr.
281
282Basically, you can think of EFL as a replacement for the GTK that SL
283viewers use. They both offer more or less a similar feature set, and
284both use more or less the same standard libraries underneath. GTK uses
285the GLib main loop, which EFL supports via wrappers. I think that EFL
286fits the SLedjHamr design goals much better though.
287
288As a bonus, lots and lots of eye candy, complete with a system for
289writing your own eye candy, and over riding the system supplied eye
290candy. So very pretty GUI skins are supported. As a special bonus,
291works great on small systems. EFL is being used for Samsungs Tizen, in
292fact half the EFL developers now work for Samsung.
293
294Flexible.
295---------
296
297The generic and modular qualities mean that people can build their own
298virtual worlds in any way they like. Use a file based asset store, or a
299database based one, or generate it on the fly using genetic algorithms,
300what ever floats your boat. Serve assets with Apache, Nginx, lighttp,
301or any other web server. Use ejabberd, or the Lua based Prosody for
302text based communications. Maybe you want to use a CMS like Drupal to
303deal with assets, users, groups, and permissions? Let your users
304manipulate their inventory, or the world, using WebDav. Use LDAP for
305users and groups. Roll your own if you don't like any of the other
306choices for any specific module. SledjHamr wont stand in your way, and
307will help you use the virtual world specific bits any way you want.
308
309Users get this flexibility to. Use GIMP or Photoshop to edit textures
310directly. Or Blender to create new meshes in world. Keep your
311inventory on your own hard drive, so you can use the tools you are
312comfortable with to manage it all. Store your inventory in "the cloud".
313Easily send things from your inventory to others without even logging in
314world, using what ever method you like. Share some of your inventory on
315social media.
316
317Create what ever sort of mash up you can bend your tools to do. Hook up
318an IRC bot to an in world group chat room. Run on online business that
319prints in world objects on 3D printers, using standard E business
320software and a virtual world as your shop front end.
321
322The virtual world is your oyster, go nuts.
323
324Er, but experimental for now.
325-----------------------------
326
327Still early days, and even though a few helped with the first design
328docs, so far all the coding work has been done by one person. So much
329to do for a virtual world system. This is why the git repo has an empty
330master branch, it's all in the experimental branch. So far only bits
331and pieces, which have only recently been tied together into one system.
332
333As experimental software, it's all up in the air as we try out different
334things, explode our own careless thought bubbles, rip things out and
335start from scratch, and in general mess with things to our hearts
336content. Nothing is set in stone yet. We write some small bit of code
337that seems at least slightly sane, or cobble something together quickly
338that might be totally insane. Then tinker with it to see what works, or
339to make it closer to our goals. Eventually we will make bits that we
340think are stable enough for actual use and move them to the master
341branch. Don't go holding your breath though.
342
343Lua has been great for quickly building prototype stuff. So far most of
344the network protocols and file formats have been based on Lua. Which
345goes against our goals, so expect that to be temporary. Specifically,
346once we get around to actually implementing the nails protocol stuff,
347some of the Lua network and file formats will get replaced be nails.
348
349The Lua GUI code is losely based on my ancient matrix-RAD stuff that was
350written in Java. The fact that I had most of it working within a week
351is a tribute to how easy Lua is. The original Java took me years to
352write. On the other hand, I'm not that happy with the syntax of the
353results, so that is likely to change.
354
355Other developers could use their own favourite languages for
356experimental work, but I highly encourage them to eventually produce C
357or Lua code depending on the requirements of that module.
358
359The specific coding style used is all over the place at the moment.
360Lots of code I wrote for other projects has been dragged in and morphed
361into our requirements, but it all came with it's own style from those
362projects. I've produced an uncrustify file for cleaning up C code, but
363not applied it to most code yet. Even that uncrustify file is subject
364to tweaking.
365
366The LuaSL code is the most stable, but that's not surprising, it's the
367first thing I wrote. On the other hand, all of the actual world
368changing LL function calls have not been written yet, there's no actual
369world code to put them in. Some one volunteered to interface LuaSL to
370OpenSim, which would pass the world changing functions to that to deal
371with, but they vanished before they started. LuaSL at least uses a
372network socket to feed the world changing functions to a world server,
373so that part can be modular and support OpenSim, the SledjHamr love
374server, or anything else.
375
376I've not settled on which 3D rendering system to use yet. The options I
377have short listed and experimented with are Irrlicht and Evas_3D.
378
379I had looked at various open source 3D render systems, including things
380like Orge, with a specific set of goals in mind. A major goal was to
381support the plethora of low powered laptops every one uses for virtual
382worlds that have bare minimum 3D support, or even none at all. Irrlicht
383has a great software renderer for people with no 3D hardware. That got
384my vote, so I experimented with it. There was no EFL integration, so I
385had to write some, which wasn't that hard. A year later, a new version
386of EFL managed to bit rot my EFL/Irrlicht integration, so I fixed that.
387But now it flickers like crazy. After much discussion with other EFL
388develpors, and some preliminary work by the Irrlicht developers, an
389experimental version of Irrlicht is underway that might integrate better
390with EFL. I've not tried it yet, but it was on my TODO.
391
392In the mean time, and so recently that it only gets released later
393today, Evas_3D was added to EFL. Evas is one of the sub libraries of
394EFL that deals with canvasses, it's a 3D mesh, camera, and light system
395designed to be part of Evas. So it has the major advantage of already
396being well integrated, it's part of EFL. I've added test code for it
397now, and no flickering. I even have test code using both Irrlicht and
398Evas_3D in the same window space. It's major disadvantage is that it's
399only a very recent project that has (until later today) not been
400released, Irrlicht is much more mature. So Evas_3D only has basic
401stuff, though it's mostly complete basic stuff. On the other hand, a
402lot of what's missing in Evas_3D I was thinking about rewriting for
403Irrlicht anyway. So far one major missing bit is Bullet Physics
404intergration, or any other physics engine. Irrlicht has Bullet, and
405another part of EFL also has Bullet. I'm hoping the authors of that
406other EFL part get together with the Evas_3D authors and figure
407something out.
408
409So right now I'm leaning more in the Evas_3D direction, and that's
410currently the default. In neither case had I written anything to
411actually create a proper virtual world, merely modifications to the
412examples provided to see how they fit in to the rest, and give me
413something to play with. I think I'll see how far I can get with Evas_3D
414before making a decision.
415
416
19The bits (or some of them). 417The bits (or some of them).
20--------------------------- 418===========================
21 419
22extantz 420extantz
23 GuiLua front end for other programs it starts 421 GuiLua front end for other programs it starts
24 3D rendering of multiple virtual worlds in tabs 422 3D rendering of multiple virtual worlds in tabs
25 including the same virtual world logged in more than once with different accounts 423 including the same virtual world logged in more than once with different accounts
424 or even just different camera views
425 could use the same thing for generating two views for stereo viewing.
26 Nails interface to virtual world backend modules. Each module converts nails commands to / from it's own network protocol. 426 Nails interface to virtual world backend modules. Each module converts nails commands to / from it's own network protocol.
27 A SledjHamr grid, which definately should be running independantly, 427 A SledjHamr grid, which definately should be running independantly,
28 so others can log on and stay on when extantz closes down 428 so others can log on and stay on when extantz closes down
@@ -36,7 +436,7 @@ extantz
36 getting OS/SL network crap from the grid 436 getting OS/SL network crap from the grid
37 translating and sending nails commands to extantz so it can update it's 3D models 437 translating and sending nails commands to extantz so it can update it's 3D models
38 including nails encapsulated GuiLUa commands to deal the with OS/SL GUI stuff 438 including nails encapsulated GuiLUa commands to deal the with OS/SL GUI stuff
39 An Open Croquet module. 439 An Open Croquet / Cobalt module.
40 importer / exporter 440 importer / exporter
41 Which basically grabs an area / selected objects from the current tabs world, 441 Which basically grabs an area / selected objects from the current tabs world,
42 then sends them as nails commands to - 442 then sends them as nails commands to -
@@ -103,6 +503,7 @@ in world object editor
103inventory browser 503inventory browser
104 Good to be separate, coz often you spend a lot of time NOT dealing with inventory. 504 Good to be separate, coz often you spend a lot of time NOT dealing with inventory.
105 On the other hand, people might have their inventory window open all the time, I do. lol 505 On the other hand, people might have their inventory window open all the time, I do. lol
506 Naturaly you could be dealing with inventory while NOT being logged in world.
106 handles local inventory as the usual nails files 507 handles local inventory as the usual nails files
107 508
108LuaSL 509LuaSL
@@ -110,6 +511,8 @@ LuaSL
110 LSL -> LuaSL compiler 511 LSL -> LuaSL compiler
111 LuaSL script runner - likely to be running thousands of scripts 512 LuaSL script runner - likely to be running thousands of scripts
112 LuaJIT to speed it up, EFL luaproc to run them as worker threads 513 LuaJIT to speed it up, EFL luaproc to run them as worker threads
514 Has since been rewritten to be an Ecore_Thread / cooperative multitasking based system, depending on compile options.
515 Should make that choosable at run time after more testing.
113 LSL constants and functions implemented as a Lua library 516 LSL constants and functions implemented as a Lua library
114 The functions want to use nails to talk to the server backend. 517 The functions want to use nails to talk to the server backend.
115 Currently just sending Lua function() call strings instead. 518 Currently just sending Lua function() call strings instead.
@@ -139,6 +542,7 @@ IM client
139 Separate coz it could also be used as a normal IM client. 542 Separate coz it could also be used as a normal IM client.
140 either sending the GuiLua to extantz, or dealing with it itself 543 either sending the GuiLua to extantz, or dealing with it itself
141 calls libpurple for the hard work 544 calls libpurple for the hard work
545 Or maybe not, I've gone off libpurple, even after naming the IM stuff purkle.
142 needs stuff for extantz / woMan to feed it IM / group / local chat stuffs from grids 546 needs stuff for extantz / woMan to feed it IM / group / local chat stuffs from grids
143 gotta be modular, perhaps same module that handles other grid stuff for extantz? 547 gotta be modular, perhaps same module that handles other grid stuff for extantz?
144 or a libpurple module for OS/SL, and have authentication credentials for those grids passed from elsewhere (woMan?) 548 or a libpurple module for OS/SL, and have authentication credentials for those grids passed from elsewhere (woMan?)