New XL Engine Release on the new blog.

Progress and Plans on the new blog.

Since the projects have moved to XLEngine.com, I will be posting links here for a while as new blog posts are made. Please follow the link above to read the latest post.

Edit – the site is back online faster then I expected. When I posted this I expected extended down-time.

This post is not directly related to DaggerXL but rather to the new XL Engine site. Since it is currently offline, I’m posting this on the blogs and old forums so people know what’s going on.

The new XL Engine site is down temporarily and will probably be down all of today and maybe even tomorrow. This is only a temporary issue that I’m attempting to get solved right now. Please be patient, the site will be back up as soon as possible.

Thanks for your patience.

As previously discussed, I’ve moved to a centralized XL Engine site and blog. So from now on I will be linking posts from the new blog from here, until everything is moved over.

Read XL-Project Move and DaggerXL Version 0.20 Update.

The DaggerXL project will soon be merged together with DarkXL as the XL Engine. While the merge itself hasn’t begun yet (see the previous post for a time table), I have begun to consolidate the communities and sites that are dealing with these projects. The forums for all these projects will now be here: xlengine.com/forums. The new site, xlengine.com will now host a new blog that will cover all the projects and the site will host news and downloads when it’s setup. I will be setting things up in the next week, after which point I’ll remove downloads from these blogs. This will also be the last blog post made specifically for this blog, though I will still put up posts here with links to the new blog so returning visitors aren’t left behind. To that end the blogs will remain operational for a while yet.

So please visit the new forums to get acquainted with them and register if you like. The site/blog isn’t fully setup yet, but as I said that will be happening within the next week.

The good news is that the new site is a properly hosted site, meaning that I have more control and things can be centralized (site, blog, forum and file hosting all on that site). This also means that things like a wiki or other features can be added later.

Version 0.20 Update

I’ll start by saying that the 0.20 version is still on track for release and will be probably come out this weekend. I’m going to be busy for the rest of the week – which is the reason for the delay, but it’s very close to completion. All the weapon types are in and working, including the different materials (with bonuses and weight changes) – with the exception of bows, which will come later once I work on the ranged combat and spells. Currently I’m working on getting all the different armor types and materials working as well.

Once version 0.20 is released, the plan is to spend some time getting the Beta for DarkXL done, which is where the changes below come in.

Project Merger

I’ve been thinking about this, off and on, for a while now. But with the DarkXL Beta coming up this is basically my “last” good chance to get this done if it’s going to be done at all. I plan on merging the DaggerXL and DarkXL projects into one engine, the XL Engine. But the engine rendering and many other aspects are different, you may say – and it’s true. But there is a lot of code that can and should be shared between the projects. For example, much of the software renderer and hardware driver abstraction layers. The in-game console. The sound system. The midi playback system. The window/OS management and input systems, the scripting system. UI scripting ability, AI scripting, and so on. As I start supporting different OS’s and rendering APIs, this will save a lot of duplicate work. I’ve been considering, way down the road – post DaggerXL beta if it happens – adding ArenaXL as a project supported by DaggerXL. With the shared code base, I can use a lot of the same code that DarkXL will use for it’s sector rendering, for example.

There are other benefits: releases that work on technology now benefit all projects, no need to “port” from one to the other. As I brushed upon before, when I add support for other platforms – I add support for all the projects. When I make releases, it will be easy to make improvements for multiple games simultaneously. Some tools and mod support can be shared across projects (for example adding the DaggerXL texture replacers support to DarkXL). And finally I can merge the communities under a single engine/site/forum so that I can be active on all the projects and not just one at a time.

So what happens to all the different sites and forums that currently exist?
DF-21 will remain a resource for Dark Forces related files and forums. However I plan on setting up a new forum, for the XL Engine, and migrate all the projects to that forum. The DaggerXL forum will remain open for quite some time, but once I get things moving I will probably focus on posting over there and this one will slowly fade away. ( As sad as that is ) For the time being, I’ll keep the blogs around but I’ll probably start putting together an XL Engine blog and mirror posts across the others. As support for more games is added in the future, this will be a more scalable solution as well.

Does this change future plans for DaggerXL, modding support or other DaggerXL specific plans?
No. Full modding support is still planned, though many of the tools may be shared with DarkXL modding, BloodXL modding and so on.

Isn’t DarkXL a sector engine and DaggerXL a true 3D polygon engine? Aren’t these incompatible?
It is true that they are different but both can exist in the same engine. DarkXL actually supports rendering models already, they are used in Dark Forces for things like bridges, Tie Fighters, The Moldy Crow and so on. The way the level geometry is rendered is indeed much different, but the engine will be able to support both sector and “free-form” polygonal geometry. This has potential implications for modding, though those will be explored later. Things like scene traversal and level geometry rendering are different, but this won’t be the first engine to support multiple methods of scene traversal and rendering. A lot of the surrounding code will still be shared, so the savings offset the cost.

So here is my plan:
1) Release version 0.20 of DaggerXL.
2) Finish the DarkXL Beta.
3) Merge DaggerXL and DarkXL under the new XL Engine, where DaggerXL and DarkXL are two games supported by the engine.
4+) Continue to work on the projects as I have been, but with everything together it’ll much easier for me to keep everything going rather then letting one or another stagnate.

The order of 2 and 3 may change. My gut instinct is to get the merge done before the Beta but completing the Beta first gets a long overdue build out sooner…, but that is beyond the scope of this topic.

Finally, I have to ask the community: Are you guys willing to do this? To move to new forums, to intermingle with Dark Forces, Outlaws and Blood fans? Of course each game gets it’s own sub-forum so it’s not complete anarchy, but you get the idea. I know it’s sudden for you guys, but it’s been on my mind for a while now. Ultimately I think this change would be an improvement for all the projects – DaggerXL included – but I would appreciate it if you guys let me know what you think.

In this post I’ll talk more about the upcoming software renderer including progress, performance and future plans. Note that this work is happening in parallel to the version 0.20 support, the software renderer won’t be available until after the next build.

As I’ve been working on the next release, I occasionally fiddle with the software renderer. Currently I’m writing it in a separate test app, to avoid integration issues until after version 0.20 is released. Until recently, screenshots would be really boring since it was all boilerplate code but I’ll go ahead and show the first screenshot now:

Statistics for images shown:

Resolution: 960×600.
Current features:

  • 32 bit renderer (8 bit will also be supported with all the palette effects)
  • Perspective correct texturing
  • 16 bit Z-Buffer (will probably be upgraded 32 bit)
  • Sub-pixel/sub-texel accuracy (no wobbling edges or textures)
  • Per polygon color, used for lighting (supports RGB, only intensity shown here)
  • Z-Fogging using a fog table.
  • Clipping.

And of course it supports 2D blits (including arbitrary scaling):

Currently performance isn’t where I want it to be for multiple reasons:

  • It’s using GDI for the final blit which can be slow. Obviously it should use DirectDraw and/or Direct 3D when available (just for the final blit though).
  • The bottleneck is the actual rasterization, in the future this will be optimized using SSE/SSE2 when available. However this won’t occur until it has all the base features.
  • SSE/SSE2 optimizations for other parts of the pipeline as needed (transform, clipping, etc.).

Some of the additional features required before integration (post-0.20):

  • Dynamic lights (using the proper radial attenuation).
  • Mipmapping (probably with mip selection done per scanline).
  • Various pipeline improvements, backface culling and so on.

Finally I plan on adding hierarchical z-buffer based occlusion culling – at the batch, polygon and maybe even span level in order to reduce overdraw to near zero. This, combined with an optimized rasterizer, should allow the software renderer to run at very high resolutions on modern CPUs. This is more practical with software rendering (or at least simpler to get right) due to the lack of latency and also the fact that it handles small batches very well, which means that rendering can be sorted from front to back at a finer granularity without hurting batch performance or state switching performance.

Ultimately the goal is 60fps at 1024×768 on a 1 GHz CPU. Once the final blit is fixed (i.e. not taking 11ms at 1680×1050 because of GDI), I’m close to that goal now but still have to make it faster using the above methods in order to also have gameplay too (i.e. collision detection, AI, combat, etc.). :)

SSE will only be used when available, potentially widening the CPU support if you run at low resolutions (i.e. 320×200 or 640×480).

Hopefully this will allow anyone with a semi-modern computer (i.e. a computer purchased within the last 11 years) to play DaggerXL at good framerates, at least with the basic feature set. In addition the hardware renderer will go through another round of optimizations, as well as general program speed improvements and loading improvements upon start up.

In this article I’ll talk about the progress towards version 0.20 as well as plans for supporting lower end systems.

Version 0.20 Progress

I’ve been continuing to work on DaggerXL version 0.20, so I’ll talk about some of the items that have recently been accomplished. If you’re following along on the DaggerXL forums, then most of this will be old news.

Dungeon Water

The first step was to figure out where the water plane height is stored in the data, since it is different for different dungeons. Once I figured that out, the next step was to render the water plane, which you can see below (thumbnails):

The next release will most likely just use the simple plane for the surface, as shown in the previous screenshots. I’ll figure out how I want to handle the surface (how faithful the default should be, that kind of thing) after the next build. In the future the water can look much cooler, but I’m still trying to focus on gameplay.

So the next step was to detect when the player was underwater and render the appropriate effects:

Finally I implemented the swimming mechanic, which uses the player’s skill to determine things such as swimming speed. In addition the swimming skill improves with use in the same way that running and climbing currently work. Finally I implemented the breath meter and drowning mechanic, though the meter was implemented after I took the above screenshots. Sound effects for entering the water, swimming and underwater ambiance have been implemented.

Object Orientation Issues

The next thing I worked on was the various object orientation issues, which cause such comical images as you see below:

These and many other weird anamolies have now been fixed for version 0.20. Most of the screenshots of various issues like these were taken by a DaggerXL forum member, Lennas – if you want to see more “bloopers,” visit the bug thread here.

Switches

Switches will work much more reliably in version 0.20. The first problem is that they were not animating correctly a lot of the time or not animating at all. These problems have been fixed. The second problem is that they often didn’t activate their target objects properly, which has also been fixed. In addition a new switch type has been found and implemented that controls the gates that you find in some dungeons.

Remaining Issues and Features

Now I am working on finish the dungeon texture region remapping, which makes dungeons in different areas of Daggerfall use different textures without having to create custom pieces for each area. This is similar to the texture remapping already done for exterior buildings, terrain and building interiors. After that is finished, the next goal is to fix some uv issues and finally flat scale/offset issues – which will fix things like floating poles and columns. On the feature side, once the bugs are dealt with, I will finish the loot system, leveling and get proper random encounters in. At this point I’ll release version 0.20 which will feature a much more robust dungeon exploring experience as well as full character advancement and leveling.

Low-End System Support

The focus of DaggerXL right now is, and will continue to be for a while, completing the “vanilla” Daggerfall gameplay. However I’m also going to start, slowly, putting in proper support for lower end hardware and optimizing the support for mid/higher end hardware. So here’s basically how I plan on tackling this issue:

1) Greater optimizations for the current renderer (of course). Things like reducing shader complexity as much as possible, reducing batch-count further and so on.

2) Support for pre-shader model 2.0 hardware with reduced effects/quality when necessary. The goal here would be to reproduce the effects as much as possible using fixed function. Any GPU should be able to run in this mode, though visual compromises will have to be made – possibly reduced exterior view distance, lighting simplification (i.e. dynamic light maps, texture projections, local tesselation + vertex lighting, etc.; not sure which one yet). – This option may be skipped in favor of option 3 below.

3) Software renderer – I’ve been wanting to play around with software rendering again lately (it’s been a while… other then some CPU raycasting experiments). The idea would be to support, with low to moderate resolutions, any system with a decent CPU(s). Older GPU’s (especially Intel) tend to have terrible and outdated drivers, so this may be a viable alternative to people with decent systems that have under-performing or old GPU’s. At low resolutions, this could also help systems with slower CPU’s since batching, CPU/GPU bus bandwidth and driver performance are no longer an issue.

Now due to the terrible drivers I mentioned, I’m actually considering skipping option #2. Basically decent Shader Model 2+ GPU (Radeon 9500+, GeForce 6+) or software. Normally I’d shy away from the complexity of adding a software renderer but there are a few compelling reasons –

* For some reason I want to tinker with software rendering. Being a hobby project, this is actually really compelling for me. :)
* By limiting the set of GPU’s to those with decent drivers and allowing a fallback for newer GPUs with bad drivers (think Intel), this should make supporting DaggerXL a little easier while making the potential audience larger.
* Support older systems without worrying about poor Driver support (at least for graphics).
* I plan on supporting multiple platforms eventually, so this would allow me to make sure that clean support is in for multiple renderers without affecting the higher level code. I.e. most of the code base shouldn’t care if it’s hardware or software.
* Wider system and possible platform support.

Of course, many of the cool new effects will be hardware only. However the software renderer should still look much cleaner then the original Daggerfall renderer. Some examples include fixing visual glitches between tiles, increased draw distance (over vanilla, less then GPU), lighting and Goraud shading on terrain, per-pixel perspective correction to make textures look cleaner, mipmapping, proper z-buffer, true color textures in 32 bit mode, etc. There will be two modes – 32 bit and 8 bit – depending on whether you want palette support or not. Texture replacements, sprite replacements and model replacements will be supported in the 32 bit software rendered mode and in the GPU mode – though performance will have to be a consideration of course.

In addition I plan on doing some limited one-time caching to speed up the initial loading times. This would happen automatically the first time the program is run or whenever the cache becomes invalidated (new version, modding, etc.). Again all automatic, but I’d like to get the load times to be much faster as well.

I’ve been working towards version 0.20 – I’d like to share the progress as it is made:

Automap

The first major feature going into version 0.20 is the dungeon automap system (and also the building interior maps – since they use a simplified version of the same system). The automap only shows those areas that you’ve already explored – like the original. Those flags are also saved in the save games. The plan is that when current saved games are loaded, all the flags are cleared and the automap will start updating from then on. Obviously this won’t be a problem when entering a dungeon for the first time or starting a new game or with new (version 0.20+) save games.

Below you can see some screenshots of the automap. These are all thumbnails, click on them to see the larger images.

There were taken before the compass and rotation were implemented:

These show rotation and the “perspective” mode. Note that the mode icon still has to change as well as the yellow “blockmap” that shows your current location and the exit/start.

As you can see the increased resolution improves the readability but that is not all. The “view distance” has been greatly increased. This allows you to zoom out and see a global view of the dungeon that you’ve explored so far. Later extended features are planned to make this even more usable such as making pieces that obscure the player arrow translucent and fogging or desaturating pieces based on how far below the player they are. The idea is to give the player a greater since of where the geometry is in space in relation to the player.

The Next Steps

As you may know, I’ve been discussing this on the DaggerXL forums a little bit… what are the next steps after version 0.20 is released when the dungeon exploration experience will be much more robust and complete? The short answer is that I will begin focusing more on the “above ground” – RPG – gameplay.

To that end the following build will focus on NPCs and the dialog system. This includes getting wandering NPC’s working again, but this time you’ll be able to talk to them as well as kill them. You will also be able to interact with static NPCs.

The immediate next step will then be to get shops, banks and other places of business operational. With all the dungeon exploring, you’ll now be able to sell off that loot and use your hard-earned gold to actually buy stuff. 😀

After that comes the spell system, including the ability to acquire new pre-made spells due to the NPC/shop functionality (which includes the Mage’s Guild non-member services such as buying pre-made spells). This will also include general projectiles such as Bows and enemy spells.

And finally the Quest System and Faction System, starting with Guilds and Guild Quests. Throughout this process I will be working on implementing missing skills, as appropriate (Mercantile when shops are implemented for example), so by this point all the skills will most likely be functional.

Beyond all this, there is still more to do. But this outlines the immediate future – and the path to DaggerXL Alpha Status. 🙂

Version 0.198 has been released, it is essentially just a bug fix build. I’ll go over the changes:

  • Fixed save crash bug in many dungeons.
  • Fixed teleport tag issues (though teleport still doesn’t function).
  • Random monsters are actually random now, though not using the correct monster sets.
  • Dungeon start and current location ID are now saved in the Save Format. Note 0.197 saves will still load.
  • Added a coordinate and location ID view – hit F11. Use this view when reporting issues.

Remember that when reporting bugs inside the dungeons, hit F11 before taking your screenshot. This has the local dungeon coordinates plus the location ID. This will allow me to teleport here in the future. Previously, once you entered a dungeon and saved – if you reload the game later it would forget where the dungeon start was. This means you’d have to find your way out of the dungeon the hard way. That has been fixed, the start point and location ID are both saved, though the old saves will still load just fine.

You can download it from the Downloads page.