Some time ago I bought a software for VJ (VJing (pronounced: VEE-JAY-ing – is a broad designation for realtime visual performance). Characteristics of VJing are the creation or manipulation of imagery in realtime through technological mediation and for an audience, in synchronization to music.
NestDrop is an ingenious software based on the visualization system produced by the programming language called Milkdrop, originally developed by Ryan Geiss in 2001.
I like Milkdrop’s beat detection system, it works well and we all know how music and lights in sync can produce pleasant moments.
So, after playing on my own for a bit I said to myself why not try to do a music and light show in Second Life, live?
It’s not as simple as it seems, especially when it comes to Second Life for video. The support is rudimentary, all you can do is apply a URL to the face of an object and start playing it (this is called MOAP, media on a prim). However, this does not guarantee at all that everyone will participate in the event, because in the case of a film, for example, each user start a playback from the beginning and so if someone arrives late they will never be aligned with the other participants, who perhaps are not even aligned with each other. We would like to have a party, not watch a movie each on a separated sofa. Give us our keyframe.
There are systems in Second Life that try to overcome this problem with scripting and other systems, but they are unreliable and complex. How to do it in a transparent, simple and economical way?
A day i visited a sim, Museum Island, where a guy with nickname Cimafilo was streaming a movie, Alice in the wonderland, and i noticed that data stream was small but fluid, in sync (also if on sim was used the parcel media and was not possible understand who and what was managing the sync), so i tried to get some infos about. Cima was using a for me unknow video format, WebM and OBS, with good results in my opinion. So I tried to create a similar system, but suited to my needs: analogous workflow, but using a dedicated server and a higher frame rate.
I can say I succeeded, the system devised can stream and sync audio and video events in Second Life using simple open source tools. Let’s see how.
As mentioned, my goal was to use a VJ program to create an audiovisual event in Second Life.
All you need is any audio player (personally I use Winamp because it’s very light, but any other player is fine) and a VJ software (in my case NestDrop, but any other is fine). The result of our audio and video mix must be able to be captured by the OBS (Open Broadcaster Software), so any program that generates video output on your PC monitor is fine.
Your desktop at the end of first part of setup: from left, Winamp produces sound, NestDrop make the visuals and OBS capture all.
I was thinking to myself, how can I send this output from OBS to Second Life with acceptable quality and in sync for anyone? Showing the video is easy, almost, there are many methods by going from a web page or a streaming service but they all lack the detail that is so important to me: sync.
However, I noticed one thing: almost all of these services use MP4 container or its variants to distribute the content. And by studying, I realized that the MP4 container does not have a sync system suitable for my purpose. I need the sync to be sent at pre-established intervals during the projection. Codecs world is a real jungle managed through trade wars.
At this point I entered a hell of questions and answers on forums, web pages, approximate and/or wrong answers, you name it. I’m also no expert on these things, and this was new territory for me. Double difficulty then.
I convinced myself along the way that the secret was in the format, and indeed it was.
Arguing with FFMPEG specs I discovered the format that’s right for me: compact, manages good audio (in Opus) and video (VP8 or VP9), open source, all browsers can view it: WebM was the trick.
Above all, a specification of webM container shoot me: Key frames SHOULD be placed at the beginning of clusters.
Exactly what I was looking for, bingo. In a few words, when the Play key is pressed in the SL viewer, each visitor gets a key frame to sync at the exact point of the actual key in the video. Yes, I want to use this!
Okay, but where do I send this WebM stream if I manage to convince OBS to do broadcasting?
Read here, search there, a server that accepts WebM is Icecast2.
Now comes the slightly complex part, because you need an Icecast video hosting, which will work like Shoutcast hosting which – if you have ever played music in Second Life – you surely already know. Or, you can implement your own server there. I obviously chose the second path both for its affordability and to fully understand how it works.
On the Web it is easy to find offers of virtual servers, even at very low prices. For me I got a VPS with 1 processor, 1 GB RAM, 100 GB hard disk. For an experiment it’s fine, in any case you can always expand. We go.
The installation of a Linux distro is always quite automated, just choose the desired distro from a drop-down menu and in a few minutes the basic server will be operational. So far so good, and I chose Ubuntu 20.04 for myself.
Preparing my VPS
In a few minutes the server is ready. I chose not to install anything apart from Icecast, but obviously the space can be used for all that you need.
Installing Icecast2 is very simple, all you need is a little familiarity with the terminal (for me, Putty) commands and your server is ready in 10 minutes. You can find dozens of tutorials, all copy/pasted from each other.
The only detail that I recommend you take care of is to open port 8000 of the server firewall, or no one will be able to connect and set your audio card setting on Windows at 48000 Hz for the audio side. You don’t need anything else.
Now test if Icecast is responding with a browser, adding :8000 to your server address.
In my case, for example: http://vps-94bab050.vps.ovh.net:8000/
The server will most likely be listening.
Pelican Village TV waiting for a mountpoint
When configuring Icecast (you will be asked to specify passwords for admin and other details), you may have a doubt about mountpoints, and the documentation isn’t very clear either: should I create them now? Or when? In reality you don’t have to do anything, OBS (or the software you use for broadcast) will create the mountpoint with the name you chose in the connection string (we see later how).
I connected OBS to server, and the connection created the mountpoint /video, as specified in connection string.
Opening the URL http://vps-94bab050.vps.ovh.net:8000/video (following the example) you can already see what you are going stream in Second Life in a browser window (if someone is streaming, or you get a “Error 404 page Not Found”). The mountpoint is dynamic, this means it is alive as long as the stream is up. When you disconnect your broadcaster, the mountpoint disappears.
To connect OBS to the Icecast server, the issue is not very simple but it can be solved.
In File > Settings > Output > Recordings, select as Output Mode Advanced.
Type: Custom Output (FFmpeg)
FFMpeg Output Type: Output to URL
File part or URL: icecast://user name:password@server URL:8000/mountpoint name
User name and password you have set installing Icecast2, remember?
Container Format and Container format Description: webm
Muxer Settings open a chapter apart. This specs and Encoder settings have do be decided carefully reading the documentation at https://trac.ffmpeg.org/wiki/EncodingForStreamingSites and https://developers.google.com/media/vp9/live-encoding Configurations depends on many factors based on your material, hardware, connection, server. For me personally the string that worked for Muxer Settings is:
content_type=video/webm cluster_size_limit=10M cluster_time_limit=5100The part most important is Encoder Settings. The string that for me worked is:
-r 30 -g 60 -quality realtime -speed 6 -threads 6 -frame-parallel 1 -tile-columns 4 -row-mt 1(For each parameter meaning and some sample configurations, refer to specs)
To start the broadcasting, push Start Recording button in OBS (we are recording to a URL).
Now what you need is to prepare the “screen”. In Second Life, associate the URL of your mountpoint with the Media texture of object you have chosen as screen and press Play on the viewer. Nothing else is needed.
Let the rave begin 🙂