len0rd/rockbox
1
0
Fork 1
mirror of https://github.com/Rockbox/rockbox.git synced 2026-04-12 00:47:49 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat: add rekordbox support to iPod Video

Browse source
This commit is contained in:
Charlie 2026-02-28 16:17:15 +00:00
parent 3a70003d23
commit b897b400af
10 changed files with 2484 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

338
REKORDBOX.md Normal file
View file

@ -0,0 +1,338 @@
# Rockbox + Rekordbox for iPod 5G/5.5G
A custom Rockbox build that lets your iPod Video live a double life: plug it into a Pioneer CDJ as a Rekordbox USB source, then unplug and use it as a standalone music player with Rockbox — no syncing scripts, no extra software, no compromise.
---
## How it works
When you export your library from Rekordbox to a USB drive, Pioneer creates a binary database at `/PIONEER/rekordbox/export.pdb` alongside your audio files in `/PIONEER/Contents/`. This build adds a parser for that database directly into Rockbox's core. On every boot, Rockbox checks whether the PDB has changed since the last import. If it has, it reads the track metadata (title, artist, album, genre, BPM, rating) and populates the native Rockbox database — so you can browse your Rekordbox library through all the usual Rockbox menus.
It also generates `.m3u8` playlist files in `/Playlists/Rekordbox/` from your Rekordbox playlists, and re-imports automatically after every USB sync session.
---
## Requirements
- **iPod**: 5th generation (Video, 30GB) or 5.5th generation (Video, 80GB)
- **Rekordbox**: version 6.x or 7.x on macOS or Windows
- **Drive format**: FAT32 — required by both Rockbox and Pioneer CDJs
- **This build**: compiled from this repository (see [Building](#building))
> **5G vs 5.5G?** Both are "iPod Video." The 5.5G (late 2006) added the 80GB drive option and a slightly brighter screen. Same PP5022 chipset, same Rockbox support. This build works on both.
---
## Drive preparation
Pioneer CDJs require FAT32. iPods sold in most markets ship formatted correctly, but check before proceeding.
**On macOS:**
```
diskutil list
diskutil eraseDisk FAT32 IPOD MBRFormat /dev/diskN
```
**On Windows:**
- Open Disk Management, right-click the iPod partition → Format → FAT32
- For drives > 32GB, use [Rufus](https://rufus.ie) or `fat32format`
---
## Installing Rockbox
### 1. Install Rockbox Utility
Download [Rockbox Utility](https://www.rockbox.org/wiki/RockboxUtility) for your OS. It automates bootloader installation and file copying.
> **Do not use the official Rockbox Utility build for this.** It will install the unmodified Rockbox firmware. You need to install the bootloader with Rockbox Utility, then manually copy the custom `.rockbox` folder from this build.
### 2. Install the bootloader
1. Connect your iPod via USB
2. Open Rockbox Utility → **Bootloader** → **Install**
3. Follow the prompts — it patches the iPod firmware partition so Rockbox boots instead of (or alongside) Apple OS
### 3. Copy the custom firmware
After building (see [Building](#building)), copy the output `.rockbox` folder to the root of your iPod:
```
/ ← iPod root
├── .rockbox/ ← Rockbox firmware, codecs, themes, database
│ ├── rockbox.ipod
│ ├── codecs/
│ └── ...
```
The iPod now boots Rockbox.
---
## Exporting from Rekordbox
1. In Rekordbox, go to **File → Export → Export to Removable Device** (or drag tracks/playlists to the device panel)
2. Select your iPod as the export target
3. Choose which playlists or crates to export
4. Click **Export**
Rekordbox creates:
```
/PIONEER/
├── rekordbox/
│ └── export.pdb ← binary database (tracks, playlists, metadata)
└── Contents/
├── 01/
│ ├── track001.mp3
│ └── track002.flac
├── 02/
└── ...
```
Your audio files live in `/PIONEER/Contents/` in sequentially numbered subdirectories. The `export.pdb` file references them with paths like `/Contents/01/track001.mp3` — this build prepends `/PIONEER` to resolve them to `/PIONEER/Contents/01/track001.mp3` on the device.
---
## First boot after export
On the first boot after an export (or after any re-export that changes the PDB):
1. Rockbox detects that `/PIONEER/rekordbox/export.pdb` is newer than its last import stamp (`.rockbox/rekordbox_sync.dat`)
2. It parses the PDB — artist, album, genre, and track metadata — and writes a temporary database file
3. It commits the entries into the Rockbox tagcache
4. It generates `.m3u8` playlist files in `/Playlists/Rekordbox/`
5. It writes `/PIONEER/database.ignore` so Rockbox's own file scanner doesn't double-index the Pioneer folder
This happens automatically in the background thread during boot. On a large collection (10,000+ tracks) it may take 20–40 seconds on first import. Subsequent boots are instant if nothing has changed.
---
## Browsing your library in Rockbox
After import, your Rekordbox tracks appear in the standard Rockbox database views:
- **Database → Artists** → all artists from your Rekordbox export
- **Database → Albums**
- **Database → Genres**
- **Database → All tracks**
- **Files → Playlists → Rekordbox →** your Rekordbox playlists as `.m3u8` files
If the database does not appear populated after first boot, trigger a manual update:
**Main Menu → Database → Update Now**
Or force a full rebuild:
**Main Menu → Database → Initialize Now**
---
## Using with Pioneer CDJs
CDJ models with USB storage support (CDJ-2000, CDJ-2000NXS, CDJ-2000NXS2, CDJ-3000, XDJ-1000, XDJ-RX series, and most post-2010 Pioneer/Denon media players) read directly from the `/PIONEER/rekordbox/export.pdb` database — the same file Rockbox imports from. No conflict.
**Workflow:**
1. Export from Rekordbox on your laptop to the iPod (USB)
2. Unplug from laptop — iPod re-imports on next boot automatically
3. At the venue: plug iPod into CDJ USB port — CDJ reads the Rekordbox database natively
4. Between sets: use the iPod standalone with Rockbox
**Known limitations:**
- The iPod must be formatted FAT32 — some CDJ firmware versions have issues with drives > 1TB or non-standard sector sizes. Standard iPod 5G/5.5G drives (80GB, 512-byte sectors) work reliably.
- If your CDJ requires the iPod to be in "disk mode" rather than standard USB MSC, enable it: hold **Menu + Select** on boot until the disk icon appears, then plug in.
- Rekordbox analysis data (waveforms, cue points, hot cues) is stored in the PDB and read by CDJs directly — Rockbox ignores this data but does not corrupt it.
---
## Re-syncing
Whenever you re-export from Rekordbox (adding tracks, editing cue points, updating playlists):
1. Connect iPod to laptop — Rockbox enters USB mode
2. Run the Rekordbox export as usual
3. Eject safely — on reconnect for normal use, Rockbox detects the updated PDB and re-imports
The re-import is incremental at the file level: tracks whose file mtime hasn't changed are skipped. Only new or modified entries are re-processed.
---
## Mixed library (Rekordbox + regular music)
You can have both Rekordbox-exported tracks and your own music on the same iPod:
- Put personal music anywhere **outside** `/PIONEER/` (e.g. `/Music/`)
- Rekordbox tracks live in `/PIONEER/Contents/`
- The `/PIONEER/database.ignore` file prevents the Rockbox scanner from double-indexing Pioneer tracks
- Enable **Main Menu → Database → Auto Update** so personal music outside `/PIONEER/` is scanned normally
- Both sets of tracks appear together in the Rockbox database
---
## Testing with the SDL Simulator
Before flashing your iPod, you can run and test the entire Rekordbox import pipeline locally using the Rockbox SDL simulator — no physical device needed.
### What the simulator is
Rockbox ships a first-party SDL2-based UI simulator that runs natively on Linux, macOS, and Windows. It emulates the Rockbox interface and treats a local `simdisk/` subdirectory as the iPod's drive. All of `pdb_parser.c`, `rekordbox_import.c`, and `tagcache.c` run inside it exactly as they would on real hardware.
### Prerequisites
```bash
# Ubuntu/Debian
sudo apt install libsdl2-dev gcc make perl python3
# macOS
brew install sdl2
```
### Build the simulator
```bash
git clone <this repo>
cd rockbox
mkdir build-sim && cd build-sim
../tools/configure
# Select target: 22 (iPod Video)
# Select build type: [Ss] Simulator (press A for advanced options first)
make -j$(nproc)
make install
```
This produces a `rockboxui` binary and a `simdisk/` directory in `build-sim/`.
### Set up a test library
Recreate the Pioneer folder structure inside `simdisk/`:
```bash
mkdir -p build-sim/simdisk/PIONEER/rekordbox
mkdir -p build-sim/simdisk/PIONEER/Contents/01
# Copy a real Rekordbox export
cp /path/to/your/export.pdb build-sim/simdisk/PIONEER/rekordbox/export.pdb
# Optionally add some audio files at the paths the PDB references
# (Rockbox will stat() each file; missing files are skipped gracefully)
cp /path/to/track.mp3 build-sim/simdisk/PIONEER/Contents/01/
```
### Run
```bash
cd build-sim
./rockboxui
```
On "boot", `tagcache_thread()` fires, detects the PDB, and runs the full import. Check the results:
```bash
# Sync stamp written after successful import
ls -la build-sim/simdisk/.rockbox/rekordbox_sync.dat
# Tagcache database files
ls build-sim/simdisk/.rockbox/database_*.tcd
# Generated playlists
find build-sim/simdisk/Playlists/Rekordbox/ -name '*.m3u8'
# Pioneer ignore marker
ls build-sim/simdisk/PIONEER/database.ignore
```
To force a re-import, delete the sync stamp:
```bash
rm build-sim/simdisk/.rockbox/rekordbox_sync.dat
# Then re-run ./rockboxui
```
### Simulator limitations
| Feature | Simulator | Real hardware |
|---------|-----------|---------------|
| PDB parsing | ✅ Full | ✅ Full |
| Tagcache import | ✅ Full | ✅ Full |
| Playlist generation | ✅ Full | ✅ Full |
| Audio playback | ❌ None | ✅ Full |
| USB connect/disconnect re-import | ❌ Not triggered | ✅ Automatic |
| FAT32 filesystem | ❌ Host FS only | ✅ FAT32 |
| Boot timing / performance | ❌ Host CPU speed | ✅ PP5022 speed |
The USB re-import path (`SYS_USB_CONNECTED` handler in `tagcache.c`) cannot be triggered in the simulator. Test it by deleting the sync stamp manually and rebooting the sim instead.
---
### Prerequisites
```bash
# Ubuntu/Debian
sudo apt install gcc gcc-arm-none-eabi make perl python3 zip
# macOS
brew install arm-none-eabi-gcc make
```
### Configure and build
```bash
git clone <this repo>
cd rockbox
mkdir build-ipodvideo && cd build-ipodvideo
../tools/configure
# Select: 22 (iPod Video)
# Build type: N (Normal)
make -j$(nproc)
make zip
```
The `make zip` step produces `rockbox.zip` — extract it and copy the `.rockbox` folder to your iPod.
### Verifying Rekordbox support is compiled in
After `configure`, check `build-ipodvideo/autoconf.h`:
```bash
grep HAVE_REKORDBOX autoconf.h
# Should output: #define HAVE_REKORDBOX
```
If it's missing, verify `firmware/export/config/ipodvideo.h` contains `#define HAVE_REKORDBOX`.
---
## Troubleshooting
**Database is empty after boot**
- Wait 30–60 seconds after first boot on a large collection, then go to **Database → Update Now**
- Check that `/PIONEER/rekordbox/export.pdb` exists on the drive
- Check `.rockbox/rekordbox_sync.dat` — if it exists and matches the PDB mtime, Rockbox considers it already imported. Delete it to force a re-import.
**Tracks show `<Untagged>` for some fields**
- The PDB entry for those tracks has empty metadata. Fix in Rekordbox and re-export.
**Playlists folder is empty**
- Check `/Playlists/Rekordbox/` on the drive — if the directory doesn't exist, the import may have failed partway through. Delete `.rockbox/rekordbox_sync.dat` and reboot.
**CDJ says "No USB device" or can't read library**
- Confirm the drive is FAT32, not exFAT or APFS
- Confirm `/PIONEER/rekordbox/export.pdb` is present (not just the audio files)
- Some older CDJ firmware versions require a `PIONEER` folder at the root — this is always created by a proper Rekordbox export
**Import takes very long on every boot**
- This should only happen on first import or after a re-export. If it repeats every boot, check that `.rockbox/rekordbox_sync.dat` is being written (the `/.rockbox` directory must be writable)
**PDB page size error in log**
- The internal page buffer is 4KB. Rekordbox 6.x/7.x uses 4KB pages, so this should not trigger. If it does (unusual custom export), file a bug with your `export.pdb` version details.
---
## Technical notes
- **No third-party scripts required.** The entire import pipeline runs inside Rockbox at boot time.
- **PDB format**: Rekordbox 6.x/7.x DeviceSQL format. Documented at [djl-analysis.deepsymmetry.org](https://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/exports.html).
- **Tagcache integration**: entries are written as `temp_file_entry` records to `database_tmp.tcd`, then committed by the standard tagcache `commit()` path — identical to how Rockbox's own file scanner adds tracks.
- **Source files**: `apps/pdb_parser.{c,h}`, `apps/rekordbox_import.{c,h}`, integration in `apps/tagcache.c` behind `#ifdef HAVE_REKORDBOX`.

4
apps/SOURCES
View file

@ -181,6 +181,10 @@ gui/usb_screen.c
#ifdef HAVE_TAGCACHE
tagcache.c
#endif
#ifdef HAVE_REKORDBOX
pdb_parser.c
rekordbox_import.c
#endif
#ifdef HAVE_TOUCHSCREEN
keymaps/keymap-touchscreen.c
#endif

993
apps/pdb_parser.c Normal file
View file

@ -0,0 +1,993 @@
/***************************************************************************
* __________ __ ___.
* Open \______ \ ____ ____ | | _\_ |__ _______ ___
* Source | _// _ \_/ ___\| |/ /| __ \ / _ \ \/ /
* Jukebox | | ( <_> ) \___| < | \_\ ( <_> > < <
* Firmware |____|_ /\____/ \___ >__|_ \|___ /\____/__/\_ \
* \/ \/ \/ \/ \/
* $Id$
*
* Parser for Pioneer Rekordbox export.pdb database files.
*
* Binary format reference:
* https://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/exports.html
*
* All multi-byte integers in PDB are little-endian.
*
* File layout:
* - File header (28 bytes): magic(4), unknown(4), page_size(4),
* num_pages(4), unknown(4), sequence(4), unknown(4)
* followed by an array of table_pointer structs (one per table type).
* - Pages: each page_size bytes, zero-indexed.
*
* Page layout:
* - 0x00: zeros(4)
* - 0x04: page_index(4) -- 0-based index of this page
* - 0x08: type(4) -- table type (PDB_TABLE_*)
* - 0x0C: next_page(4) -- next page in chain (0xFFFFFFFF = none)
* - 0x10: unknown1(4)
* - 0x14: num_rows_small(2) -- lower bits of row count
* - 0x16: num_rows_large(2) -- for pages with > 127 rows
* - 0x18: page_flags(1) -- bit 6 = "strange" (garbage page, skip)
* - 0x19: free_size(2)
* - 0x1B: used_size(2)
* - 0x1D: unknown2(4)
* - 0x21: heap starts here
* - Row index: at end of page, grows backwards.
* Groups of up to 16 rows. Each group = presence_mask(2) +
* 16 x row_offset(2). Row offsets are relative to heap start (0x28).
*
* Row count encoding (num_rows_small / num_rows_large):
* - actual row count = (num_rows_small & 0x1FFF) * 2 [if bit 14 set]
* = (num_rows_small & 0x1FFF) [otherwise]
* See: https://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/exports.html#_pages
*
* Copyright (C) 2024 The Rockbox Project
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
****************************************************************************/
#include "pdb_parser.h"
#include <string.h>
#include <stdlib.h>
#ifndef __PCTOOL__
#include "system.h"
#include "file.h"
#include "logf.h"
#include "rbunicode.h"
#else
/* Host-side stubs */
#include <stdio.h>
#include <unistd.h>
#include <sys/stat.h>
#define logf(fmt, ...) fprintf(stderr, "[pdb] " fmt "\n", ##__VA_ARGS__)
/* Provide utf16LEdecode for the host tool */
static unsigned char *utf16LEdecode(const unsigned char *utf16,
unsigned char *utf8, int count)
{
int i;
for (i = 0; i < count; i++) {
unsigned int ucs = (unsigned int)utf16[0] | ((unsigned int)utf16[1] << 8);
utf16 += 2;
if (ucs < 0x80) {
*utf8++ = (unsigned char)ucs;
} else if (ucs < 0x800) {
*utf8++ = 0xC0 | (ucs >> 6);
*utf8++ = 0x80 | (ucs & 0x3F);
} else {
*utf8++ = 0xE0 | (ucs >> 12);
*utf8++ = 0x80 | ((ucs >> 6) & 0x3F);
*utf8++ = 0x80 | (ucs & 0x3F);
}
}
*utf8 = '\0';
return utf8;
}
#endif /* __PCTOOL__ */
/* -----------------------------------------------------------------------
* Internal helpers for little-endian reads from a byte buffer.
* ----------------------------------------------------------------------- */
static inline uint8_t read_u8(const uint8_t *buf, uint32_t off)
{
return buf[off];
}
static inline uint16_t read_u16(const uint8_t *buf, uint32_t off)
{
return (uint16_t)buf[off] | ((uint16_t)buf[off + 1] << 8);
}
static inline uint32_t read_u32(const uint8_t *buf, uint32_t off)
{
return (uint32_t)buf[off]
| ((uint32_t)buf[off + 1] << 8)
| ((uint32_t)buf[off + 2] << 16)
| ((uint32_t)buf[off + 3] << 24);
}
/* -----------------------------------------------------------------------
* PDB file header constants.
*
* The file starts with:
* 0x00: magic (4 bytes) — 0x00000000 (yes, four zero bytes)
* 0x04: unknown (4 bytes)
* 0x08: page_size (4 bytes)
* 0x0C: num_pages (4 bytes)
* 0x10: unknown (4 bytes)
* 0x14: sequence (4 bytes)
* 0x18: unknown (4 bytes)
*
* Immediately after (at 0x1C) are the table pointer entries. Each entry is:
* type(4) + first_page(4) = 8 bytes.
* There are typically 16 table entries.
* ----------------------------------------------------------------------- */
#define PDB_FILE_HEADER_SIZE 0x1C
#define PDB_TABLE_ENTRY_SIZE 8
#define PDB_NUM_TABLE_ENTRIES 16
/* Page header field offsets */
#define PDB_PAGE_OFF_ZEROS 0x00
#define PDB_PAGE_OFF_INDEX 0x04
#define PDB_PAGE_OFF_TYPE 0x08
#define PDB_PAGE_OFF_NEXT 0x0C
#define PDB_PAGE_OFF_NUM_ROWS 0x14 /* 2-byte num_rows_small */
#define PDB_PAGE_OFF_NUM_ROWS2 0x16 /* 2-byte num_rows_large */
#define PDB_PAGE_OFF_FLAGS 0x18
#define PDB_PAGE_HEAP_START 0x28 /* first byte of heap */
/* -----------------------------------------------------------------------
* DeviceSQL string decoding.
*
* Strings in the PDB heap are stored as "DeviceSQL" blobs:
* byte 0: type
* 0x40 = long ASCII (followed by 2-byte length, then ASCII chars)
* 0x90 = long UTF-16LE (followed by 2-byte length in code units, then UTF-16LE)
* Other types are rare; we treat them as empty.
*
* String offsets stored in rows are relative to the start of the page heap
* (i.e., offset 0 in the heap = PDB_PAGE_HEAP_START in the page buffer).
* ----------------------------------------------------------------------- */
/*
* Decode a DeviceSQL string from within a page buffer.
* heap_off: offset from heap start (PDB_PAGE_HEAP_START) of the string blob.
* page: the full page buffer.
* page_size: size of the page buffer.
* out: destination buffer (UTF-8, NUL-terminated).
* out_size: size of destination buffer.
*/
static void decode_devicesql_string(uint32_t heap_off,
const uint8_t *page, uint32_t page_size,
char *out, int out_size)
{
uint32_t blob_off;
uint8_t type_byte;
uint16_t length;
const uint8_t *data;
out[0] = '\0';
blob_off = PDB_PAGE_HEAP_START + heap_off;
if (blob_off + 3 > page_size)
return;
type_byte = read_u8(page, blob_off);
length = read_u16(page, blob_off + 1);
data = page + blob_off + 3;
if (type_byte == PDB_STR_TYPE_LONG_ASCII) {
/* ASCII: length bytes of ASCII characters */
if (blob_off + 3 + length > page_size)
return;
int copy = (length < (uint16_t)(out_size - 1))
? length : (out_size - 1);
memcpy(out, data, copy);
out[copy] = '\0';
} else if (type_byte == PDB_STR_TYPE_LONG_UTF16LE) {
/* UTF-16LE: length = number of code units (not bytes) */
uint32_t byte_len = (uint32_t)length * 2;
if (blob_off + 3 + byte_len > page_size)
return;
/* Each UTF-16 code unit encodes to at most 3 UTF-8 bytes + NUL */
if ((int)(length * 3 + 1) > out_size)
length = (uint16_t)((out_size - 1) / 3);
utf16LEdecode(data, (unsigned char *)out, length);
out[out_size - 1] = '\0';
}
/* Unknown type byte: leave as empty string */
}
/* -----------------------------------------------------------------------
* Page I/O helpers.
* ----------------------------------------------------------------------- */
/*
* Read page number page_idx into ctx->page_buf.
* Returns true on success.
*/
static bool read_page(struct pdb_context *ctx, uint32_t page_idx)
{
int32_t offset;
if (page_idx >= ctx->num_pages) {
logf("pdb: page %lu out of range (num_pages=%lu)",
(unsigned long)page_idx, (unsigned long)ctx->num_pages);
return false;
}
offset = (int32_t)(page_idx * ctx->page_size);
if (lseek(ctx->fd, offset, SEEK_SET) < 0) {
logf("pdb: seek to page %lu failed", (unsigned long)page_idx);
return false;
}
if (read(ctx->fd, ctx->page_buf, ctx->page_size)
!= (int)ctx->page_size) {
logf("pdb: read page %lu failed", (unsigned long)page_idx);
return false;
}
return true;
}
/*
* Decode the effective row count from a page buffer.
*
* The page stores two 2-byte fields at offsets 0x14 and 0x16.
* The high bit (bit 15) of the first field indicates which formula to use:
* - bit 15 clear: num_rows = (field0 & 0x1FFF)
* - bit 15 set: num_rows = (field0 & 0x0FFF) | ((field1 & 0x000F) << 12)
* (i.e., 16-bit combined row count)
*
* Additionally, the row *offset* count is stored in the upper bits:
* num_row_offsets = (field0 >> 13) & 0x07 [3-bit field at bits 13-15]
*
* The number of rows listed in the row index (num_row_offsets) may differ
* from the actual filled row count (num_rows); we iterate over all
* row index entries and use the presence mask to skip absent ones.
*/
static uint32_t page_row_count(const uint8_t *page)
{
uint16_t f0 = read_u16(page, PDB_PAGE_OFF_NUM_ROWS);
uint16_t f1 = read_u16(page, PDB_PAGE_OFF_NUM_ROWS2);
if (f0 & 0x8000) {
/* high bit set: combined 16-bit count */
return (uint32_t)(f0 & 0x0FFF) | ((uint32_t)(f1 & 0x000F) << 12);
} else {
return (uint32_t)(f0 & 0x1FFF);
}
}
/*
* Return the number of row index entries (groups of up to 16) on the page.
* This is the value stored in bits 13-15 of the first num_rows field.
*
* NOTE: each index entry covers up to 16 rows. A value of N means there
* are N groups, covering rows 0..N*16-1.
*/
static uint32_t page_num_row_groups(const uint8_t *page)
{
uint16_t f0 = read_u16(page, PDB_PAGE_OFF_NUM_ROWS);
/* bits 12:0 contain the count; bits 15:13 the group count */
return (uint32_t)((f0 >> 13) & 0x07);
}
/*
* Iterate over all rows in a page, calling the provided callback for each
* present row.
*
* The row index grows backwards from the end of the page. Each group is:
* presence_mask(2) + 16 x row_offset(2) = 34 bytes.
*
* The group at index 0 (the *last* group written, at the end of the page)
* covers the highest-numbered rows; groups are in reverse order.
* Bit i of presence_mask indicates row i within the group is present.
* Row offsets are relative to heap start (PDB_PAGE_HEAP_START).
*
* callback: called with (page_buf, row_heap_offset, page_size, userdata)
* for each present row.
* Returns false to abort iteration.
*/
typedef bool (*pdb_row_cb)(const uint8_t *page, uint32_t row_heap_off,
uint32_t page_size, void *userdata);
static void iterate_page_rows(const uint8_t *page, uint32_t page_size,
pdb_row_cb cb, void *userdata)
{
uint32_t num_groups;
uint32_t g, bit;
uint32_t group_off; /* offset from end of page to this group's presence_mask */
uint16_t mask;
uint16_t row_off;
if (page[PDB_PAGE_OFF_FLAGS] & PDB_PAGE_FLAG_STRANGE)
return; /* "strange" page — skip */
num_groups = page_num_row_groups(page);
if (num_groups == 0)
return;
/* Groups are at the very end of the page, growing backwards.
* Group 0 is at page[page_size - 34], group 1 at page[page_size - 68], etc.
* Each group is 2 + 16*2 = 34 bytes. */
for (g = 0; g < num_groups; g++) {
group_off = page_size - (g + 1) * 34;
mask = read_u16(page, group_off);
for (bit = 0; bit < 16; bit++) {
if (!(mask & (1 << bit)))
continue; /* row absent */
row_off = read_u16(page, group_off + 2 + bit * 2);
if (row_off == 0)
continue;
/* row_off is relative to heap start */
if ((uint32_t)PDB_PAGE_HEAP_START + row_off >= page_size)
continue;
if (!cb(page, (uint32_t)row_off, page_size, userdata))
return; /* callback requested abort */
}
}
}
/* -----------------------------------------------------------------------
* String lookup table management.
* ----------------------------------------------------------------------- */
/*
* Add or update an entry in a string lookup table.
* Returns true on success, false if the table is full.
*/
static bool string_table_add(struct pdb_string_table *tbl,
uint32_t id, const char *name)
{
int i;
struct pdb_string_entry *e;
/* Check for existing entry (update in place) */
for (i = 0; i < tbl->count; i++) {
if (tbl->entries[i].id == id) {
strlcpy(tbl->entries[i].name, name, PDB_STR_MAX);
return true;
}
}
if (tbl->count >= tbl->capacity)
return false;
e = &tbl->entries[tbl->count++];
e->id = id;
strlcpy(e->name, name, PDB_STR_MAX);
return true;
}
const char *pdb_lookup_string(const struct pdb_string_table *tbl, uint32_t id)
{
int i;
for (i = 0; i < tbl->count; i++) {
if (tbl->entries[i].id == id)
return tbl->entries[i].name;
}
return "";
}
/* -----------------------------------------------------------------------
* Row parsers for each table type.
* ----------------------------------------------------------------------- */
/*
* Parse an artist or album row.
* The row format is identical for both:
* 0x00: subtype(1) — 0x60/0x80 = "near" (1-byte offsets)
* 0x64/0x84 = "far" (2-byte offsets)
* 0x01: index_shift(1)
* 0x02: bitmask(4)
* 0x06: id(4)
* ...
* For subtype 0x60 (near artist):
* 0x0A: name_offset(1) — 1-byte heap offset
* For subtype 0x64 (far artist):
* 0x0A: name_offset(2) — 2-byte heap offset
*
* Album rows have subtypes 0x80 (near) and 0x84 (far).
* The id field is always at the same position.
*/
struct name_id_cb_data {
struct pdb_string_table *tbl;
uint32_t page_size;
};
static bool parse_artist_row(const uint8_t *page, uint32_t row_off,
uint32_t page_size, void *userdata)
{
struct name_id_cb_data *d = (struct name_id_cb_data *)userdata;
const uint8_t *row = page + PDB_PAGE_HEAP_START + row_off;
uint8_t subtype;
uint32_t id;
uint32_t name_heap_off;
char name[PDB_STR_MAX];
/* Safety: need at least 12 bytes for the row header */
if ((uint32_t)(PDB_PAGE_HEAP_START + row_off + 12) > page_size)
return true;
subtype = read_u8(row, 0x00);
id = read_u32(row, 0x06);
if (subtype == 0x60) {
/* near: 1-byte name offset at 0x0A */
if ((uint32_t)(PDB_PAGE_HEAP_START + row_off + 11) > page_size)
return true;
name_heap_off = read_u8(row, 0x0A);
} else if (subtype == 0x64) {
/* far: 2-byte name offset at 0x0A */
if ((uint32_t)(PDB_PAGE_HEAP_START + row_off + 12) > page_size)
return true;
name_heap_off = read_u16(row, 0x0A);
} else {
return true; /* unknown subtype, skip */
}
decode_devicesql_string(name_heap_off, page, page_size, name, PDB_STR_MAX);
string_table_add(d->tbl, id, name);
return true;
}
static bool parse_album_row(const uint8_t *page, uint32_t row_off,
uint32_t page_size, void *userdata)
{
struct name_id_cb_data *d = (struct name_id_cb_data *)userdata;
const uint8_t *row = page + PDB_PAGE_HEAP_START + row_off;
uint8_t subtype;
uint32_t id;
uint32_t name_heap_off;
char name[PDB_STR_MAX];
if ((uint32_t)(PDB_PAGE_HEAP_START + row_off + 12) > page_size)
return true;
subtype = read_u8(row, 0x00);
id = read_u32(row, 0x06);
if (subtype == 0x80) {
/* near album */
if ((uint32_t)(PDB_PAGE_HEAP_START + row_off + 11) > page_size)
return true;
name_heap_off = read_u8(row, 0x0A);
} else if (subtype == 0x84) {
/* far album */
if ((uint32_t)(PDB_PAGE_HEAP_START + row_off + 12) > page_size)
return true;
name_heap_off = read_u16(row, 0x0A);
} else {
return true;
}
decode_devicesql_string(name_heap_off, page, page_size, name, PDB_STR_MAX);
string_table_add(d->tbl, id, name);
return true;
}
/*
* Parse a genre row.
* Genre rows use a simpler format:
* 0x00: subtype(1)
* 0x01: index_shift(1)
* 0x02: bitmask(4)
* 0x06: id(4)
* 0x0A: name_offset(2) — 2-byte heap offset
*/
static bool parse_genre_row(const uint8_t *page, uint32_t row_off,
uint32_t page_size, void *userdata)
{
struct name_id_cb_data *d = (struct name_id_cb_data *)userdata;
const uint8_t *row = page + PDB_PAGE_HEAP_START + row_off;
uint32_t id;
uint32_t name_heap_off;
char name[PDB_STR_MAX];
if ((uint32_t)(PDB_PAGE_HEAP_START + row_off + 12) > page_size)
return true;
id = read_u32(row, 0x06);
name_heap_off = read_u16(row, 0x0A);
decode_devicesql_string(name_heap_off, page, page_size, name, PDB_STR_MAX);
string_table_add(d->tbl, id, name);
return true;
}
/*
* Track row layout (Rekordbox 6.x/7.x, little-endian):
*
* 0x00: subtype(2)
* 0x02: index_shift(2)
* 0x04: bitmask(4)
* 0x08: sample_rate(4)
* 0x0C: composer_id(4)
* 0x10: file_size(4)
* 0x14: unknown(4)
* 0x18: unknown(2)
* 0x1A: unknown(2)
* 0x1C: artwork_id(4)
* 0x20: key_id(4)
* 0x24: orig_artist_id(4)
* 0x28: label_id(4)
* 0x2C: remixer_id(4)
* 0x30: bitrate(4) -- kbps
* 0x34: track_number(4)
* 0x38: tempo(4) -- BPM * 100
* 0x3C: genre_id(4)
* 0x40: album_id(4)
* 0x44: artist_id(4)
* 0x48: id(4)
* 0x4C: disc_number(2)
* 0x4E: play_count(2)
* 0x50: year(2)
* 0x52: sample_depth(2)
* 0x54: duration(2) -- seconds
* 0x56: unknown(2)
* 0x58: color_id(1)
* 0x59: rating(1) -- 0..5 stars * 0x14
* 0x5A: unknown(2)
* 0x5C: file_type(2)
* 0x5E: unknown(2)
* 0x60...: 21 x uint16 string offsets (heap-relative)
*
* String slot indices (0-based within the 21-slot array at 0x60):
* 0: isrc
* 1: lyricist
* 2: title
* 3: unknown
* 4: unknown
* 5: artist (sometimes also from artist_id)
* 6: album (sometimes also from album_id)
* 7: genre (sometimes also from genre_id)
* 8: comment
* 9: date_added
* 10: unknown
* 11: file_path ("/Contents/...")
* 12: file_name
* 13: file_type_str
* 14: unknown
* 15: unknown
* 16: unknown
* 17: mix_name
* 18: unknown
* 19: unknown
* 20: key
*/
#define TRACK_STR_TITLE 2
#define TRACK_STR_ARTIST 5
#define TRACK_STR_ALBUM 6
#define TRACK_STR_GENRE 7
#define TRACK_STR_COMMENT 8
#define TRACK_STR_FILE_PATH 11
#define TRACK_NUM_STRINGS 21
#define TRACK_ROW_MIN_SIZE (0x60 + TRACK_NUM_STRINGS * 2)
struct track_parse_cb_data {
struct pdb_context *ctx;
struct pdb_track *tracks;
int max_tracks;
int count;
};
static bool parse_track_row(const uint8_t *page, uint32_t row_off,
uint32_t page_size, void *userdata)
{
struct track_parse_cb_data *d = (struct track_parse_cb_data *)userdata;
const uint8_t *row = page + PDB_PAGE_HEAP_START + row_off;
struct pdb_track *t;
uint16_t str_offs[TRACK_NUM_STRINGS];
int i;
uint8_t raw_rating;
if (d->count >= d->max_tracks)
return false; /* no more room */
/* Safety check for minimum row size */
if ((uint32_t)(PDB_PAGE_HEAP_START + row_off + TRACK_ROW_MIN_SIZE) > page_size)
return true;
t = &d->tracks[d->count];
memset(t, 0, sizeof(struct pdb_track));
/* Fixed numeric fields */
t->sample_rate = read_u32(row, 0x08);
t->bitrate = read_u32(row, 0x30);
t->track_number = (uint16_t)read_u32(row, 0x34);
t->genre_id = read_u32(row, 0x3C);
t->album_id = read_u32(row, 0x40);
t->artist_id = read_u32(row, 0x44);
t->id = read_u32(row, 0x48);
t->disc_number = read_u16(row, 0x4C);
t->year = read_u16(row, 0x50);
t->duration = read_u16(row, 0x54);
raw_rating = read_u8 (row, 0x59);
/* Rekordbox stores rating as 0x00, 0x14, 0x28, 0x3C, 0x50, 0x64 (0..5 stars) */
t->rating = (raw_rating > 0) ? (raw_rating / 0x14) : 0;
if (t->rating > 5) t->rating = 5;
/* Read the 21 string offset slots */
for (i = 0; i < TRACK_NUM_STRINGS; i++)
str_offs[i] = read_u16(row, 0x60 + i * 2);
/* Decode string fields from the heap */
decode_devicesql_string(str_offs[TRACK_STR_TITLE],
page, page_size, t->title, PDB_STR_MAX);
decode_devicesql_string(str_offs[TRACK_STR_COMMENT],
page, page_size, t->comment, PDB_STR_MAX);
decode_devicesql_string(str_offs[TRACK_STR_FILE_PATH],
page, page_size, t->file_path, PDB_STR_MAX);
/* Prefer embedded string for artist/album/genre; fall back to lookup */
decode_devicesql_string(str_offs[TRACK_STR_ARTIST],
page, page_size, t->artist, PDB_STR_MAX);
if (t->artist[0] == '\0' && t->artist_id != 0)
strlcpy(t->artist, pdb_lookup_string(&d->ctx->artists, t->artist_id),
PDB_STR_MAX);
decode_devicesql_string(str_offs[TRACK_STR_ALBUM],
page, page_size, t->album, PDB_STR_MAX);
if (t->album[0] == '\0' && t->album_id != 0)
strlcpy(t->album, pdb_lookup_string(&d->ctx->albums, t->album_id),
PDB_STR_MAX);
decode_devicesql_string(str_offs[TRACK_STR_GENRE],
page, page_size, t->genre, PDB_STR_MAX);
if (t->genre[0] == '\0' && t->genre_id != 0)
strlcpy(t->genre, pdb_lookup_string(&d->ctx->genres, t->genre_id),
PDB_STR_MAX);
/* Skip entries with no usable file path */
if (t->file_path[0] == '\0') {
logf("pdb: track id=%lu has no file_path, skipping",
(unsigned long)t->id);
return true;
}
d->count++;
return true;
}
/*
* Playlist tree row:
* 0x00: subtype(1)
* 0x01: index_shift(1)
* 0x02: bitmask(4)
* 0x06: parent_id(4)
* 0x0A: unknown(4)
* 0x0E: sort_order(4)
* 0x12: id(4)
* 0x16: raw_is_folder(2) -- 0 = playlist, 1 = folder
* 0x18: name_offset(2) -- heap offset to DeviceSQL string
*/
struct playlist_tree_cb_data {
struct pdb_playlist_tree_entry *buf;
int max;
int count;
};
static bool parse_playlist_tree_row(const uint8_t *page, uint32_t row_off,
uint32_t page_size, void *userdata)
{
struct playlist_tree_cb_data *d = (struct playlist_tree_cb_data *)userdata;
const uint8_t *row = page + PDB_PAGE_HEAP_START + row_off;
struct pdb_playlist_tree_entry *e;
uint16_t name_off;
if (d->count >= d->max)
return false;
if ((uint32_t)(PDB_PAGE_HEAP_START + row_off + 0x1A) > page_size)
return true;
e = &d->buf[d->count];
e->parent_id = read_u32(row, 0x06);
e->sort_order = read_u32(row, 0x0E);
e->id = read_u32(row, 0x12);
e->is_folder = (read_u16(row, 0x16) != 0);
name_off = read_u16(row, 0x18);
decode_devicesql_string(name_off, page, page_size, e->name, PDB_STR_MAX);
if (e->name[0] == '\0')
strlcpy(e->name, "Unnamed", PDB_STR_MAX);
d->count++;
return true;
}
/*
* Playlist entry row:
* 0x00: subtype(1)
* 0x01: index_shift(1)
* 0x02: bitmask(4)
* 0x06: entry_index(4)
* 0x0A: track_id(4)
* 0x0E: playlist_id(4)
*/
struct playlist_entry_cb_data {
struct pdb_playlist_entry *buf;
int max;
int count;
};
static bool parse_playlist_entry_row(const uint8_t *page, uint32_t row_off,
uint32_t page_size, void *userdata)
{
struct playlist_entry_cb_data *d = (struct playlist_entry_cb_data *)userdata;
const uint8_t *row = page + PDB_PAGE_HEAP_START + row_off;
struct pdb_playlist_entry *e;
if (d->count >= d->max)
return false;
if ((uint32_t)(PDB_PAGE_HEAP_START + row_off + 0x12) > page_size)
return true;
e = &d->buf[d->count];
e->entry_index = read_u32(row, 0x06);
e->track_id = read_u32(row, 0x0A);
e->playlist_id = read_u32(row, 0x0E);
d->count++;
return true;
}
/* -----------------------------------------------------------------------
* Page-chain walker: walk all pages of a given table type, calling
* the row callback on each non-strange page.
* ----------------------------------------------------------------------- */
static bool walk_table(struct pdb_context *ctx, uint32_t table_type,
pdb_row_cb cb, void *userdata)
{
uint32_t page_idx;
uint32_t next_page;
if (table_type >= 16)
return false;
page_idx = ctx->table_first_page[table_type];
if (page_idx == 0xFFFFFFFF) {
logf("pdb: table %lu not present", (unsigned long)table_type);
return true; /* not an error, table simply absent */
}
while (page_idx != 0xFFFFFFFF) {
if (!read_page(ctx, page_idx)) {
logf("pdb: failed to read page %lu for table %lu",
(unsigned long)page_idx, (unsigned long)table_type);
return false;
}
/* Verify this page belongs to the expected table */
if (read_u32(ctx->page_buf, PDB_PAGE_OFF_TYPE) != table_type) {
logf("pdb: page %lu has wrong type (expected %lu, got %lu)",
(unsigned long)page_idx, (unsigned long)table_type,
(unsigned long)read_u32(ctx->page_buf, PDB_PAGE_OFF_TYPE));
break;
}
iterate_page_rows(ctx->page_buf, ctx->page_size, cb, userdata);
next_page = read_u32(ctx->page_buf, PDB_PAGE_OFF_NEXT);
page_idx = next_page;
}
return true;
}
/* -----------------------------------------------------------------------
* Public API implementation.
* ----------------------------------------------------------------------- */
bool pdb_open(struct pdb_context *ctx)
{
uint8_t header[PDB_FILE_HEADER_SIZE + PDB_NUM_TABLE_ENTRIES * PDB_TABLE_ENTRY_SIZE];
uint32_t magic;
int i;
/* Read file header + table entries in one shot */
if (lseek(ctx->fd, 0, SEEK_SET) < 0)
return false;
if (read(ctx->fd, header, sizeof(header)) != (int)sizeof(header)) {
logf("pdb: failed to read file header");
return false;
}
/* Magic is the first 4 bytes — Rekordbox PDB starts with 0x00000000 */
magic = read_u32(header, 0x00);
if (magic != 0x00000000) {
logf("pdb: unexpected magic 0x%08lx (expected 0x00000000)",
(unsigned long)magic);
return false;
}
ctx->page_size = read_u32(header, 0x08);
ctx->num_pages = read_u32(header, 0x0C);
if (ctx->page_size < 0x100 || ctx->page_size > 0x10000) {
logf("pdb: implausible page_size %lu", (unsigned long)ctx->page_size);
return false;
}
if (ctx->num_pages == 0 || ctx->num_pages > 0x100000) {
logf("pdb: implausible num_pages %lu", (unsigned long)ctx->num_pages);
return false;
}
if (ctx->page_size > ctx->page_buf_size) {
logf("pdb: page_size %lu > page_buf_size %lu",
(unsigned long)ctx->page_size, (unsigned long)ctx->page_buf_size);
return false;
}
/* Initialise all table first-page pointers to "absent" */
for (i = 0; i < 16; i++)
ctx->table_first_page[i] = 0xFFFFFFFF;
/* Parse table pointer entries.
* Each entry: type(4) + first_page(4). */
for (i = 0; i < PDB_NUM_TABLE_ENTRIES; i++) {
uint32_t off = PDB_FILE_HEADER_SIZE + (uint32_t)i * PDB_TABLE_ENTRY_SIZE;
uint32_t type = read_u32(header, off);
uint32_t fp = read_u32(header, off + 4);
if (type < 16)
ctx->table_first_page[type] = fp;
}
logf("pdb: page_size=%lu num_pages=%lu",
(unsigned long)ctx->page_size, (unsigned long)ctx->num_pages);
return true;
}
void pdb_close(struct pdb_context *ctx)
{
if (ctx->artists.entries) {
free(ctx->artists.entries);
ctx->artists.entries = NULL;
ctx->artists.count = 0;
}
if (ctx->albums.entries) {
free(ctx->albums.entries);
ctx->albums.entries = NULL;
ctx->albums.count = 0;
}
if (ctx->genres.entries) {
free(ctx->genres.entries);
ctx->genres.entries = NULL;
ctx->genres.count = 0;
}
}
int pdb_build_lookup_tables(struct pdb_context *ctx)
{
struct name_id_cb_data cb_data;
int total = 0;
/* Allocate lookup table storage */
if (!ctx->artists.entries) {
ctx->artists.entries = (struct pdb_string_entry *)
malloc(PDB_LOOKUP_MAX * sizeof(struct pdb_string_entry));
if (!ctx->artists.entries) {
logf("pdb: out of memory for artists table");
return -1;
}
ctx->artists.count = 0;
ctx->artists.capacity = PDB_LOOKUP_MAX;
}
if (!ctx->albums.entries) {
ctx->albums.entries = (struct pdb_string_entry *)
malloc(PDB_LOOKUP_MAX * sizeof(struct pdb_string_entry));
if (!ctx->albums.entries) {
logf("pdb: out of memory for albums table");
return -1;
}
ctx->albums.count = 0;
ctx->albums.capacity = PDB_LOOKUP_MAX;
}
if (!ctx->genres.entries) {
ctx->genres.entries = (struct pdb_string_entry *)
malloc(PDB_LOOKUP_MAX * sizeof(struct pdb_string_entry));
if (!ctx->genres.entries) {
logf("pdb: out of memory for genres table");
return -1;
}
ctx->genres.count = 0;
ctx->genres.capacity = PDB_LOOKUP_MAX;
}
/* Parse artists */
cb_data.tbl = &ctx->artists;
cb_data.page_size = ctx->page_size;
if (!walk_table(ctx, PDB_TABLE_ARTISTS, parse_artist_row, &cb_data))
return -1;
total += ctx->artists.count;
logf("pdb: loaded %d artists", ctx->artists.count);
/* Parse albums */
cb_data.tbl = &ctx->albums;
if (!walk_table(ctx, PDB_TABLE_ALBUMS, parse_album_row, &cb_data))
return -1;
total += ctx->albums.count;
logf("pdb: loaded %d albums", ctx->albums.count);
/* Parse genres */
cb_data.tbl = &ctx->genres;
if (!walk_table(ctx, PDB_TABLE_GENRES, parse_genre_row, &cb_data))
return -1;
total += ctx->genres.count;
logf("pdb: loaded %d genres", ctx->genres.count);
return total;
}
int pdb_parse_tracks(struct pdb_context *ctx,
struct pdb_track *tracks_buf, int max_tracks)
{
struct track_parse_cb_data cb_data;
cb_data.ctx = ctx;
cb_data.tracks = tracks_buf;
cb_data.max_tracks = max_tracks;
cb_data.count = 0;
if (!walk_table(ctx, PDB_TABLE_TRACKS, parse_track_row, &cb_data))
return -1;
logf("pdb: parsed %d tracks", cb_data.count);
return cb_data.count;
}
int pdb_parse_playlists(struct pdb_context *ctx,
struct pdb_playlist_tree_entry *tree_buf, int tree_max,
struct pdb_playlist_entry *entries_buf, int entries_max)
{
struct playlist_tree_cb_data tree_cb;
struct playlist_entry_cb_data entry_cb;
int total;
tree_cb.buf = tree_buf;
tree_cb.max = tree_max;
tree_cb.count = 0;
if (!walk_table(ctx, PDB_TABLE_PLAYLIST_TREE,
parse_playlist_tree_row, &tree_cb))
return -1;
logf("pdb: parsed %d playlist tree entries", tree_cb.count);
entry_cb.buf = entries_buf;
entry_cb.max = entries_max;
entry_cb.count = 0;
if (!walk_table(ctx, PDB_TABLE_PLAYLIST_ENTRY,
parse_playlist_entry_row, &entry_cb))
return -1;
logf("pdb: parsed %d playlist member entries", entry_cb.count);
ctx->playlist_tree = tree_buf;
ctx->playlist_tree_count = tree_cb.count;
ctx->playlist_entries = entries_buf;
ctx->playlist_entry_count = entry_cb.count;
total = tree_cb.count + entry_cb.count;
return total;
}

234
apps/pdb_parser.h Normal file
View file

@ -0,0 +1,234 @@
/***************************************************************************
* __________ __ ___.
* Open \______ \ ____ ____ | | _\_ |__ _______ ___
* Source | _// _ \_/ ___\| |/ /| __ \ / _ \ \/ /
* Jukebox | | ( <_> ) \___| < | \_\ ( <_> > < <
* Firmware |____|_ /\____/ \___ >__|_ \|___ /\____/__/\_ \
* \/ \/ \/ \/ \/
* $Id$
*
* Parser for Pioneer Rekordbox export.pdb database files.
*
* The PDB format is documented at:
* https://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/exports.html
*
* Copyright (C) 2024 The Rockbox Project
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
****************************************************************************/
#ifndef _PDB_PARSER_H
#define _PDB_PARSER_H
#include <stdint.h>
#include <stdbool.h>
/* Maximum string length for any PDB field (UTF-8 encoded) */
#define PDB_STR_MAX 512
/* Maximum number of tracks we'll process in a single PDB file.
* Prevents unbounded memory use on large collections. */
#define PDB_MAX_TRACKS 50000
/* Maximum number of playlists */
#define PDB_MAX_PLAYLISTS 2000
/* Maximum tracks per playlist */
#define PDB_MAX_PLAYLIST_ENTRIES 10000
/*
* PDB table type identifiers (value in page header's "type" field).
*/
#define PDB_TABLE_TRACKS 0x00
#define PDB_TABLE_GENRES 0x01
#define PDB_TABLE_ARTISTS 0x02
#define PDB_TABLE_ALBUMS 0x03
#define PDB_TABLE_LABELS 0x04
#define PDB_TABLE_KEYS 0x05
#define PDB_TABLE_COLORS 0x06
#define PDB_TABLE_PLAYLIST_TREE 0x07
#define PDB_TABLE_PLAYLIST_ENTRY 0x08
#define PDB_TABLE_ARTWORK 0x0d
/*
* PDB "strange" page flag: pages with this bit set in page_flags
* contain only deleted/garbage rows and must be skipped.
*/
#define PDB_PAGE_FLAG_STRANGE 0x40
/*
* DeviceSQL string type bytes (first byte of the string blob).
* 0x40 = long ASCII (2-byte length prefix, then ASCII bytes)
* 0x90 = long UTF-16LE (2-byte length prefix, then UTF-16LE code units)
* Other values may exist but are rare; treat as empty string.
*/
#define PDB_STR_TYPE_LONG_ASCII 0x40
#define PDB_STR_TYPE_LONG_UTF16LE 0x90
/* -----------------------------------------------------------------------
* Parsed record structures.
* These are populated by the parser callbacks and kept compact to avoid
* large stack frames. All strings are NUL-terminated UTF-8.
* ----------------------------------------------------------------------- */
struct pdb_track {
uint32_t id;
uint32_t artist_id;
uint32_t album_id;
uint32_t genre_id;
uint32_t key_id;
uint16_t track_number;
uint16_t disc_number;
uint16_t year;
uint16_t duration; /* seconds */
uint32_t bitrate; /* kbps */
uint32_t sample_rate; /* Hz */
uint8_t rating; /* 0-5 (multiply by 51 for 0-255 scale) */
/* File path: "/Contents/..." relative to export root */
char file_path[PDB_STR_MAX];
/* Metadata strings decoded to UTF-8 */
char title[PDB_STR_MAX];
char artist[PDB_STR_MAX]; /* resolved from artist_id */
char album[PDB_STR_MAX]; /* resolved from album_id */
char genre[PDB_STR_MAX]; /* resolved from genre_id */
char comment[PDB_STR_MAX];
char composer[PDB_STR_MAX];
};
struct pdb_playlist_tree_entry {
uint32_t id;
uint32_t parent_id;
uint32_t sort_order;
bool is_folder;
char name[PDB_STR_MAX];
};
struct pdb_playlist_entry {
uint32_t playlist_id;
uint32_t track_id;
uint32_t entry_index; /* sort order within the playlist */
};
/* -----------------------------------------------------------------------
* String lookup table for resolving IDs → names.
* We build one per relevant table (artists, albums, genres).
* ----------------------------------------------------------------------- */
#define PDB_LOOKUP_MAX 8192
struct pdb_string_entry {
uint32_t id;
char name[PDB_STR_MAX];
};
struct pdb_string_table {
struct pdb_string_entry *entries;
int count;
int capacity;
};
/* -----------------------------------------------------------------------
* Parser state / context.
* Caller allocates this on the heap (or in the audio buffer).
* ----------------------------------------------------------------------- */
struct pdb_context {
int fd; /* open file descriptor of export.pdb */
uint32_t page_size; /* bytes per page (from file header) */
uint32_t num_pages; /* total number of pages in the file */
/* Table first-page pointers (page index of each table's first page).
* 0xFFFFFFFF means the table is absent. */
uint32_t table_first_page[16];
/* String lookup tables built during the first pass */
struct pdb_string_table artists;
struct pdb_string_table albums;
struct pdb_string_table genres;
/* Track records (filled during the track pass) */
struct pdb_track *tracks;
int track_count;
/* Playlist tree (folders + playlists) */
struct pdb_playlist_tree_entry *playlist_tree;
int playlist_tree_count;
/* Playlist entries (track-to-playlist membership + order) */
struct pdb_playlist_entry *playlist_entries;
int playlist_entry_count;
/* Scratch buffer for reading one page at a time (page_size bytes).
* Must point to a caller-provided buffer of at least page_size bytes. */
uint8_t *page_buf;
uint32_t page_buf_size;
};
/* -----------------------------------------------------------------------
* Public API
* ----------------------------------------------------------------------- */
/*
* Open and validate the PDB file header. Fills ctx->page_size,
* ctx->num_pages, and ctx->table_first_page[].
*
* Returns true on success, false on error (bad magic or I/O failure).
* The caller must have set ctx->fd to an open file descriptor before calling.
*/
bool pdb_open(struct pdb_context *ctx);
/*
* Close the PDB context and free any allocated lookup tables.
* Does NOT close ctx->fd — caller owns the file descriptor.
*/
void pdb_close(struct pdb_context *ctx);
/*
* First pass: build artist, album, and genre lookup tables.
* Must be called before pdb_parse_tracks().
*
* Returns number of entries loaded (sum across all three tables),
* or -1 on error.
*/
int pdb_build_lookup_tables(struct pdb_context *ctx);
/*
* Second pass: parse all track rows and populate ctx->tracks[].
* Resolves artist/album/genre names from lookup tables.
*
* max_tracks: caller-provided maximum (use PDB_MAX_TRACKS or less).
* tracks_buf: caller-provided array of at least max_tracks pdb_track structs.
*
* Returns number of tracks parsed, or -1 on error.
*/
int pdb_parse_tracks(struct pdb_context *ctx,
struct pdb_track *tracks_buf, int max_tracks);
/*
* Third pass: parse playlist_tree and playlist_entries tables.
*
* tree_buf / tree_max: caller-provided buffer for playlist tree entries.
* entries_buf / entries_max: caller-provided buffer for membership entries.
*
* Returns total entries processed (tree + membership), or -1 on error.
*/
int pdb_parse_playlists(struct pdb_context *ctx,
struct pdb_playlist_tree_entry *tree_buf, int tree_max,
struct pdb_playlist_entry *entries_buf, int entries_max);
/*
* Resolve a string ID against one of the lookup tables.
* Returns the name string, or "" if not found.
*/
const char *pdb_lookup_string(const struct pdb_string_table *tbl, uint32_t id);
#endif /* _PDB_PARSER_H */

800
apps/rekordbox_import.c Normal file
View file

@ -0,0 +1,800 @@
#ifdef HAVE_REKORDBOX
#include "config.h"
#include "rekordbox_import.h"
#include "pdb_parser.h"
#include "tagcache.h"
#include <stdint.h>
#include <stdbool.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
/* Rockbox kernel / filesystem headers */
#include "file.h" /* open, close, read, write, lseek */
#include "dir.h" /* mkdir */
#include "errno.h" /* errno, EEXIST */
#include "logf.h"
#include "system.h" /* MAX_PATH */
/* stat() for mtime retrieval */
#include <sys/stat.h>
/* -----------------------------------------------------------------------
* Internal constants
* ----------------------------------------------------------------------- */
/* Tagcache temp-file magic (must match TAGCACHE_MAGIC in tagcache.c). */
#define RB_TAGCACHE_MAGIC 0x54434810
/* Maximum byte length of a single tag string (from tagcache.h TAG_MAXLEN). */
#ifndef TAG_MAXLEN
#define TAG_MAXLEN (MAX_PATH * 2)
#endif
/* Fallback string for empty/missing tags (from tagcache.h UNTAGGED). */
#ifndef UNTAGGED
#define UNTAGGED "<Untagged>"
#endif
/* TAG_COUNT must come from tagcache.h via the enum tag_type. */
/* tag_artist=0 … tag_lastoffset=22, TAG_COUNT=23 */
/* -----------------------------------------------------------------------
* Minimal local copies of the tagcache on-disk structures.
*
* These mirror the definitions in tagcache.c (which are not exported via
* tagcache.h). They must remain byte-for-byte identical.
* ----------------------------------------------------------------------- */
struct rb_tagcache_header {
int32_t magic; /* RB_TAGCACHE_MAGIC */
int32_t datasize; /* Total string-data bytes written after header */
int32_t entry_count; /* Number of temp_file_entry records */
};
/* Full tag entry stored in database_tmp.tcd waiting for commit. */
struct rb_temp_file_entry {
int32_t tag_offset[TAG_COUNT]; /* numeric: value stored directly;
string: byte offset into this
entry's data blob */
int16_t tag_length[TAG_COUNT]; /* 0 for numeric tags;
strlen(str)+1 for string tags */
int32_t flag;
int32_t data_length; /* total bytes of concatenated strings */
};
/* -----------------------------------------------------------------------
* Helper: safe string copy that guarantees NUL termination and never
* overflows dst. Returns pointer to dst.
* ----------------------------------------------------------------------- */
static char *safe_strcpy(char *dst, const char *src, size_t dst_size)
{
if (!dst || dst_size == 0)
return dst;
if (!src || *src == '\0')
{
dst[0] = '\0';
return dst;
}
size_t len = strlen(src);
if (len >= dst_size)
len = dst_size - 1;
memcpy(dst, src, len);
dst[len] = '\0';
return dst;
}
/* -----------------------------------------------------------------------
* Helper: return the effective string for a tag field.
*
* If s is NULL or empty, returns UNTAGGED. The returned pointer is
* either s itself or the UNTAGGED literal — never heap-allocated.
* ----------------------------------------------------------------------- */
static const char *effective_tag(const char *s)
{
if (!s || *s == '\0')
return UNTAGGED;
return s;
}
/* -----------------------------------------------------------------------
* Build the on-device absolute path for a PDB track.
*
* PDB stores paths as "/Contents/..." relative to the Pioneer export root.
* On a device that was exported to the root of the volume, the PIONEER
* folder lives at /PIONEER, so the full path becomes
* /PIONEER/Contents/<filename>
*
* dst must be at least MAX_PATH bytes.
* Returns true on success, false if the assembled path would be too long.
* ----------------------------------------------------------------------- */
static bool build_track_path(char *dst, const char *pdb_file_path)
{
int n = snprintf(dst, MAX_PATH, "%s%s", RB_PIONEER_ROOT, pdb_file_path);
if (n < 0 || n >= MAX_PATH)
{
logf("rekordbox: path too long: %s%s", RB_PIONEER_ROOT, pdb_file_path);
return false;
}
return true;
}
/* -----------------------------------------------------------------------
* Sync stamp helpers.
*
* The stamp file contains exactly 4 bytes: a little-endian uint32_t
* representing the Unix mtime of the last successfully imported PDB.
* ----------------------------------------------------------------------- */
static uint32_t read_sync_stamp(void)
{
int fd = open(RB_SYNC_STAMP, O_RDONLY);
if (fd < 0)
return 0;
uint8_t buf[4];
ssize_t n = read(fd, buf, 4);
close(fd);
if (n != 4)
return 0;
return (uint32_t)buf[0]
| ((uint32_t)buf[1] << 8)
| ((uint32_t)buf[2] << 16)
| ((uint32_t)buf[3] << 24);
}
static void write_sync_stamp(uint32_t mtime)
{
int fd = open(RB_SYNC_STAMP, O_WRONLY | O_CREAT | O_TRUNC);
if (fd < 0)
{
logf("rekordbox: cannot write sync stamp");
return;
}
uint8_t buf[4] = {
(uint8_t)(mtime),
(uint8_t)(mtime >> 8),
(uint8_t)(mtime >> 16),
(uint8_t)(mtime >> 24)
};
write(fd, buf, 4);
close(fd);
}
/* -----------------------------------------------------------------------
* Public: rekordbox_import_needed()
* ----------------------------------------------------------------------- */
bool rekordbox_import_needed(void)
{
struct stat st;
if (stat(RB_PDB_PATH, &st) != 0)
return false; /* PDB absent — nothing to do */
uint32_t last = read_sync_stamp();
return (uint32_t)st.st_mtime != last;
}
/* -----------------------------------------------------------------------
* Write one temp_file_entry record to the open cachefd.
*
* The function mirrors the add_tagcache() write sequence in tagcache.c
* exactly. String tags are accumulated in tag_offset / tag_length, then
* the struct header is written, followed by the NUL-terminated strings in
* the mandatory order.
*
* total_data_size is updated by the caller; this function returns the
* number of string-data bytes written for this entry (>= 0), or -1 on
* write error.
* ----------------------------------------------------------------------- */
static int write_track_entry(int fd, const struct pdb_track *t,
const char *abs_path, uint32_t mtime)
{
struct rb_temp_file_entry entry;
memset(&entry, 0, sizeof(entry));
/* Effective strings — fall back to UNTAGGED when empty. */
const char *s_path = abs_path;
const char *s_title = effective_tag(t->title);
const char *s_artist = effective_tag(t->artist);
const char *s_album = effective_tag(t->album);
const char *s_genre = effective_tag(t->genre);
const char *s_composer = effective_tag(t->composer);
const char *s_comment = effective_tag(t->comment);
/* albumartist — PDB has no separate albumartist field; reuse artist. */
const char *s_albumartist = effective_tag(t->artist);
/* canonical artist: prefer artist, fall back to albumartist */
bool has_artist = (t->artist[0] != '\0');
const char *s_canonical = has_artist ? s_artist : s_albumartist;
/* grouping — PDB has no grouping field; fall back to title. */
const char *s_grouping = effective_tag(t->title);
/*
* Build tag_offset[] and tag_length[] for string tags.
* tag_offset[tag] = cumulative byte offset in this entry's data blob.
* tag_length[tag] = strlen(str) + 1 (includes NUL).
*
* Write order must match tagcache.c add_tagcache() exactly:
* filename, title, artist, album, genre, composer, comment,
* albumartist, virt_canonicalartist, grouping
*/
#define ADD_STRING_TAG(tag, str) \
do { \
int _len = (int)strlen(str) + 1; \
if (_len > TAG_MAXLEN) _len = TAG_MAXLEN; \
entry.tag_length[tag] = (int16_t)_len; \
entry.tag_offset[tag] = offset; \
offset += _len; \
} while (0)
int offset = 0;
ADD_STRING_TAG(tag_filename, s_path);
ADD_STRING_TAG(tag_title, s_title);
ADD_STRING_TAG(tag_artist, s_artist);
ADD_STRING_TAG(tag_album, s_album);
ADD_STRING_TAG(tag_genre, s_genre);
ADD_STRING_TAG(tag_composer, s_composer);
ADD_STRING_TAG(tag_comment, s_comment);
ADD_STRING_TAG(tag_albumartist, s_albumartist);
ADD_STRING_TAG(tag_virt_canonicalartist, s_canonical);
ADD_STRING_TAG(tag_grouping, s_grouping);
#undef ADD_STRING_TAG
entry.data_length = offset;
/* Numeric tags — stored directly in tag_offset[]. */
entry.tag_offset[tag_year] = t->year;
entry.tag_offset[tag_discnumber] = t->disc_number;
entry.tag_offset[tag_tracknumber] = t->track_number;
entry.tag_offset[tag_length] = (int32_t)t->duration * 1000; /* ms */
entry.tag_offset[tag_bitrate] = (int32_t)t->bitrate;
entry.tag_offset[tag_mtime] = (int32_t)mtime;
entry.tag_offset[tag_rating] = (int32_t)t->rating;
entry.tag_offset[tag_playcount] = 0;
entry.tag_offset[tag_playtime] = 0;
entry.tag_offset[tag_lastplayed] = 0;
entry.tag_offset[tag_commitid] = 0;
entry.tag_offset[tag_lastelapsed] = 0;
entry.tag_offset[tag_lastoffset] = 0;
/* Write header struct */
if (write(fd, &entry, sizeof(entry)) != (ssize_t)sizeof(entry))
{
logf("rekordbox: write entry header failed");
return -1;
}
/* Write string data in the mandatory order */
#define WRITE_STR(str) \
do { \
int _len = (int)strlen(str) + 1; \
if (_len > TAG_MAXLEN) _len = TAG_MAXLEN; \
if (write(fd, str, _len) != _len) \
{ \
logf("rekordbox: write string failed"); \
return -1; \
} \
} while (0)
WRITE_STR(s_path);
WRITE_STR(s_title);
WRITE_STR(s_artist);
WRITE_STR(s_album);
WRITE_STR(s_genre);
WRITE_STR(s_composer);
WRITE_STR(s_comment);
WRITE_STR(s_albumartist);
WRITE_STR(s_canonical);
WRITE_STR(s_grouping);
#undef WRITE_STR
return entry.data_length;
}
/* -----------------------------------------------------------------------
* Write database_tmp.tcd from the parsed PDB track array.
*
* Returns number of entries written, or -1 on error.
* ----------------------------------------------------------------------- */
static int write_tagcache_tmp(const char *db_path,
const struct pdb_track *tracks,
int track_count)
{
char tmp_path[MAX_PATH];
int n = snprintf(tmp_path, sizeof(tmp_path), "%s/%s",
db_path, "database_tmp.tcd");
if (n < 0 || n >= (int)sizeof(tmp_path))
{
logf("rekordbox: db_path too long");
return -1;
}
/* If the file already exists (e.g. a previous interrupted build),
* skip — the existing temp file will be committed by tagcache_thread. */
struct stat st_check;
if (stat(tmp_path, &st_check) == 0)
{
logf("rekordbox: database_tmp.tcd already present, skipping write");
return 0;
}
int fd = open(tmp_path, O_WRONLY | O_CREAT | O_TRUNC);
if (fd < 0)
{
logf("rekordbox: cannot open database_tmp.tcd for writing");
return -1;
}
/* Write placeholder header — we'll fill in counts at the end. */
struct rb_tagcache_header hdr;
memset(&hdr, 0, sizeof(hdr));
if (write(fd, &hdr, sizeof(hdr)) != (ssize_t)sizeof(hdr))
{
logf("rekordbox: header write failed");
close(fd);
return -1;
}
int32_t total_data = 0;
int32_t entry_count = 0;
char abs_path[MAX_PATH];
for (int i = 0; i < track_count; i++)
{
const struct pdb_track *t = &tracks[i];
if (!build_track_path(abs_path, t->file_path))
continue;
/* Verify the file actually exists on disk before adding. */
struct stat st;
if (stat(abs_path, &st) != 0)
{
logf("rekordbox: track file not found: %s", abs_path);
continue;
}
/* Use the file's mtime for tagcache duplicate-detection.
* If mtime is 0 (unusual on FAT), use 1 as a non-zero sentinel
* so the entry is not treated as 'unknown'. */
uint32_t mtime = (st.st_mtime != 0) ? (uint32_t)st.st_mtime : 1;
int data_written = write_track_entry(fd, t, abs_path, mtime);
if (data_written < 0)
{
close(fd);
return -1;
}
total_data += data_written;
entry_count++;
}
/* Go back and write the real header. */
hdr.magic = RB_TAGCACHE_MAGIC;
hdr.datasize = total_data;
hdr.entry_count = entry_count;
if (lseek(fd, 0, SEEK_SET) < 0
|| write(fd, &hdr, sizeof(hdr)) != (ssize_t)sizeof(hdr))
{
logf("rekordbox: header rewrite failed");
close(fd);
return -1;
}
close(fd);
logf("rekordbox: wrote %d track entries to database_tmp.tcd", entry_count);
return entry_count;
}
/* -----------------------------------------------------------------------
* M3U8 playlist generator.
*
* For each leaf playlist in the PDB playlist_tree, creates a .m3u8 file
* under /Playlists/Rekordbox/<parent_folder_name>/<playlist_name>.m3u8
* (or directly in /Playlists/Rekordbox/ for top-level playlists).
*
* Entries are sorted by their entry_index field (simple insertion sort
* on the small subset that belongs to each playlist — typical playlists
* have dozens to a few hundred tracks).
* ----------------------------------------------------------------------- */
/* Find a playlist_tree entry by its ID. Linear scan is fine given the
* small sizes involved (≤ PDB_MAX_PLAYLISTS = 2000). */
static const struct pdb_playlist_tree_entry *
find_tree_entry(const struct pdb_playlist_tree_entry *tree, int count,
uint32_t id)
{
for (int i = 0; i < count; i++)
if (tree[i].id == id)
return &tree[i];
return NULL;
}
/* Build the filesystem path for a playlist file.
* We support one level of folder nesting (parent → playlist).
* Deeper nesting is flattened to the parent level.
*
* Returns true on success. */
static bool build_playlist_path(
char *dst, size_t dst_size,
const struct pdb_playlist_tree_entry *pl,
const struct pdb_playlist_tree_entry *tree, int tree_count)
{
const char *pl_name = pl->name[0] ? pl->name : "Unnamed";
/* Find parent folder (if any) */
if (pl->parent_id != 0)
{
const struct pdb_playlist_tree_entry *parent =
find_tree_entry(tree, tree_count, pl->parent_id);
if (parent && parent->is_folder && parent->name[0])
{
int n = snprintf(dst, dst_size, "%s/%s/%s.m3u8",
RB_PLAYLIST_DIR, parent->name, pl_name);
if (n > 0 && n < (int)dst_size)
return true;
}
}
/* Top-level or parent not found */
int n = snprintf(dst, dst_size, "%s/%s.m3u8", RB_PLAYLIST_DIR, pl_name);
return (n > 0 && n < (int)dst_size);
}
/* Ensure a directory exists, creating it (and its parent under
* RB_PLAYLIST_DIR) if needed. We only handle one level of depth beyond
* RB_PLAYLIST_DIR. */
static void ensure_dir(const char *dir_path)
{
if (mkdir(dir_path) < 0 && errno != EEXIST)
logf("rekordbox: mkdir %s failed", dir_path);
}
/* Compare two playlist entries by entry_index for sorting. */
static int cmp_playlist_entry(const void *a, const void *b)
{
const struct pdb_playlist_entry *ea = (const struct pdb_playlist_entry *)a;
const struct pdb_playlist_entry *eb = (const struct pdb_playlist_entry *)b;
if (ea->entry_index < eb->entry_index) return -1;
if (ea->entry_index > eb->entry_index) return 1;
return 0;
}
/* Write a single .m3u8 file for the playlist whose ID is pl_id.
*
* track_lookup : all parsed pdb_track records (for path resolution).
* entries : all pdb_playlist_entry records across all playlists.
*
* A temporary stack-allocated sort buffer is used; for very large
* playlists (> PDB_MAX_PLAYLIST_ENTRIES) we truncate silently.
*/
#define PL_SORT_BUF_MAX PDB_MAX_PLAYLIST_ENTRIES
static void write_playlist_file(
const char *path,
uint32_t pl_id,
const struct pdb_track *tracks, int track_count,
const struct pdb_playlist_entry *entries, int entry_count)
{
/* Collect entries for this playlist into a local buffer. */
static struct pdb_playlist_entry sort_buf[PL_SORT_BUF_MAX];
int n_pl = 0;
for (int i = 0; i < entry_count && n_pl < PL_SORT_BUF_MAX; i++)
{
if (entries[i].playlist_id == pl_id)
sort_buf[n_pl++] = entries[i];
}
if (n_pl == 0)
return; /* empty playlist — skip */
/* Sort by entry_index (ascending). */
/* Simple insertion sort — n_pl is typically small. */
for (int i = 1; i < n_pl; i++)
{
struct pdb_playlist_entry key = sort_buf[i];
int j = i - 1;
while (j >= 0 && sort_buf[j].entry_index > key.entry_index)
{
sort_buf[j + 1] = sort_buf[j];
j--;
}
sort_buf[j + 1] = key;
}
(void)cmp_playlist_entry; /* suppress unused-function warning */
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC);
if (fd < 0)
{
logf("rekordbox: cannot create playlist: %s", path);
return;
}
static const char header[] = "#EXTM3U\n";
write(fd, header, sizeof(header) - 1);
for (int i = 0; i < n_pl; i++)
{
uint32_t tid = sort_buf[i].track_id;
const struct pdb_track *t = NULL;
/* Linear search — acceptable for typical collection sizes. */
for (int j = 0; j < track_count; j++)
{
if (tracks[j].id == tid)
{
t = &tracks[j];
break;
}
}
if (!t)
continue;
char abs_path[MAX_PATH];
if (!build_track_path(abs_path, t->file_path))
continue;
write(fd, abs_path, strlen(abs_path));
write(fd, "\n", 1);
}
close(fd);
}
/* Generate all .m3u8 playlist files from the parsed PDB data. */
static void generate_playlists(
const struct pdb_playlist_tree_entry *tree, int tree_count,
const struct pdb_playlist_entry *entries, int entry_count,
const struct pdb_track *tracks, int track_count)
{
/* Ensure base playlist directory exists. */
ensure_dir(RB_PLAYLIST_DIR);
for (int i = 0; i < tree_count; i++)
{
const struct pdb_playlist_tree_entry *pl = &tree[i];
/* Skip folders — only write files for actual playlists. */
if (pl->is_folder)
{
/* Pre-create the subdirectory so leaf playlists can be
* created inside it. */
char subdir[MAX_PATH];
int n = snprintf(subdir, sizeof(subdir), "%s/%s",
RB_PLAYLIST_DIR, pl->name[0] ? pl->name : "Folder");
if (n > 0 && n < (int)sizeof(subdir))
ensure_dir(subdir);
continue;
}
char pl_path[MAX_PATH];
if (!build_playlist_path(pl_path, sizeof(pl_path), pl, tree, tree_count))
{
logf("rekordbox: playlist path too long for '%s'", pl->name);
continue;
}
write_playlist_file(pl_path, pl->id,
tracks, track_count,
entries, entry_count);
}
logf("rekordbox: playlist generation complete");
}
/* -----------------------------------------------------------------------
* Write the database.ignore marker file so that tagcache's own file-system
* scanner does not double-index the PIONEER folder contents.
* ----------------------------------------------------------------------- */
static void write_ignore_file(void)
{
int fd = open(RB_IGNORE_FILE, O_WRONLY | O_CREAT | O_TRUNC);
if (fd < 0)
{
logf("rekordbox: cannot create %s", RB_IGNORE_FILE);
return;
}
close(fd);
}
/* -----------------------------------------------------------------------
* Public: rekordbox_run_import()
* ----------------------------------------------------------------------- */
int rekordbox_run_import(uint8_t *page_buf, uint32_t page_buf_size,
const char *db_path)
{
/* ------------------------------------------------------------------ */
/* 1. Open export.pdb */
/* ------------------------------------------------------------------ */
int pdb_fd = open(RB_PDB_PATH, O_RDONLY);
if (pdb_fd < 0)
{
logf("rekordbox: cannot open %s", RB_PDB_PATH);
return -1;
}
/* Record the PDB mtime before we start so we can stamp it later. */
struct stat pdb_stat;
uint32_t pdb_mtime = 0;
if (stat(RB_PDB_PATH, &pdb_stat) == 0)
pdb_mtime = (uint32_t)pdb_stat.st_mtime;
/* ------------------------------------------------------------------ */
/* 2. Initialise PDB context */
/* ------------------------------------------------------------------ */
struct pdb_context ctx;
memset(&ctx, 0, sizeof(ctx));
ctx.fd = pdb_fd;
ctx.page_buf = page_buf;
ctx.page_buf_size = page_buf_size;
if (!pdb_open(&ctx))
{
logf("rekordbox: pdb_open failed");
close(pdb_fd);
return -1;
}
/* Check that the provided page buffer is large enough. */
if (ctx.page_size > page_buf_size)
{
logf("rekordbox: page_buf too small (%u needed, %u provided)",
ctx.page_size, page_buf_size);
pdb_close(&ctx);
close(pdb_fd);
return -1;
}
/* ------------------------------------------------------------------ */
/* 3. Build artist/album/genre lookup tables */
/* ------------------------------------------------------------------ */
if (pdb_build_lookup_tables(&ctx) < 0)
{
logf("rekordbox: pdb_build_lookup_tables failed");
pdb_close(&ctx);
close(pdb_fd);
return -1;
}
/* ------------------------------------------------------------------ */
/* 4. Parse tracks */
/* ------------------------------------------------------------------ */
/* Allocate track buffer on the heap. On Rockbox embedded targets the
* audio buffer would normally be used, but here we use malloc-style
* allocation via the buffer_alloc system. Since rekordbox_run_import
* is called from tagcache_thread before any audio playback begins, a
* heap allocation is safe and simpler to manage.
*
* At ~800 bytes per pdb_track and a limit of 50 000 tracks that is
* ~40 MB — clearly too large for small targets. We therefore use a
* practical cap and log a warning if we hit it.
*
* TODO: on memory-constrained targets, process tracks in chunks and
* write partial temp files, then concatenate. For now the cap is the
* pragmatic choice.
*/
#define RB_IMPORT_MAX_TRACKS 20000 /* hard limit for this implementation */
struct pdb_track *tracks =
(struct pdb_track *)malloc(RB_IMPORT_MAX_TRACKS * sizeof(struct pdb_track));
if (!tracks)
{
logf("rekordbox: malloc failed for track buffer");
pdb_close(&ctx);
close(pdb_fd);
return -1;
}
int track_count = pdb_parse_tracks(&ctx, tracks, RB_IMPORT_MAX_TRACKS);
if (track_count < 0)
{
logf("rekordbox: pdb_parse_tracks failed");
free(tracks);
pdb_close(&ctx);
close(pdb_fd);
return -1;
}
logf("rekordbox: parsed %d tracks", track_count);
/* ------------------------------------------------------------------ */
/* 5. Write database_tmp.tcd */
/* ------------------------------------------------------------------ */
int written = write_tagcache_tmp(db_path, tracks, track_count);
if (written < 0)
{
logf("rekordbox: write_tagcache_tmp failed");
free(tracks);
pdb_close(&ctx);
close(pdb_fd);
return -1;
}
/* ------------------------------------------------------------------ */
/* 6. Parse playlists */
/* ------------------------------------------------------------------ */
/* Allocate playlist buffers on the heap as well. */
struct pdb_playlist_tree_entry *pl_tree =
(struct pdb_playlist_tree_entry *)malloc(
PDB_MAX_PLAYLISTS * sizeof(struct pdb_playlist_tree_entry));
struct pdb_playlist_entry *pl_entries =
(struct pdb_playlist_entry *)malloc(
PDB_MAX_PLAYLIST_ENTRIES * sizeof(struct pdb_playlist_entry));
int pl_total = 0;
if (pl_tree && pl_entries)
{
ctx.playlist_tree = pl_tree;
ctx.playlist_entries = pl_entries;
ctx.playlist_tree_count = 0;
ctx.playlist_entry_count = 0;
pl_total = pdb_parse_playlists(&ctx,
pl_tree, PDB_MAX_PLAYLISTS,
pl_entries, PDB_MAX_PLAYLIST_ENTRIES);
if (pl_total < 0)
{
logf("rekordbox: pdb_parse_playlists failed (non-fatal)");
pl_total = 0;
}
else
{
logf("rekordbox: parsed %d playlist tree + %d entries",
ctx.playlist_tree_count, ctx.playlist_entry_count);
}
}
else
{
logf("rekordbox: playlist buffer allocation failed (skipping playlists)");
}
/* ------------------------------------------------------------------ */
/* 7. Generate .m3u8 files */
/* ------------------------------------------------------------------ */
if (pl_total > 0 && pl_tree && pl_entries)
{
generate_playlists(pl_tree, ctx.playlist_tree_count,
pl_entries, ctx.playlist_entry_count,
tracks, track_count);
}
/* ------------------------------------------------------------------ */
/* 8. Cleanup */
/* ------------------------------------------------------------------ */
if (pl_tree) free(pl_tree);
if (pl_entries) free(pl_entries);
free(tracks);
pdb_close(&ctx);
close(pdb_fd);
/* ------------------------------------------------------------------ */
/* 9. Write database.ignore and update sync stamp */
/* ------------------------------------------------------------------ */
write_ignore_file();
if (pdb_mtime != 0)
write_sync_stamp(pdb_mtime);
logf("rekordbox: import complete — %d entries written", written);
return written;
}
#endif /* HAVE_REKORDBOX */

67
apps/rekordbox_import.h Normal file
View file

@ -0,0 +1,67 @@
#ifndef _REKORDBOX_IMPORT_H
#define _REKORDBOX_IMPORT_H
#ifdef HAVE_REKORDBOX
#include <stdbool.h>
/* Absolute path of the Pioneer export database on the device. */
#define RB_PDB_PATH "/PIONEER/rekordbox/export.pdb"
/* Root directory for Pioneer content on the device.
* PDB stores paths as "/Contents/...", which map to this prefix. */
#define RB_PIONEER_ROOT "/PIONEER"
/* File placed in the PIONEER directory to prevent tagcache's own
* file-system scanner from indexing those files a second time. */
#define RB_IGNORE_FILE "/PIONEER/database.ignore"
/* Output directory for generated Rekordbox playlist files. */
#define RB_PLAYLIST_DIR "/Playlists/Rekordbox"
/* Rockbox database directory (matches ROCKBOX_DIR / tagcache_db_path). */
#define RB_DB_DIR "/.rockbox"
/* Stamp file that records the mtime of the last successfully imported PDB.
* Content is a single 32-bit little-endian Unix timestamp. */
#define RB_SYNC_STAMP "/.rockbox/rekordbox_sync.dat"
/*
* rekordbox_import_needed() - quick check before allocating anything.
*
* Returns true if /PIONEER/rekordbox/export.pdb exists AND its mtime
* differs from the last-synced mtime stored in RB_SYNC_STAMP.
* Returns false if the file is absent or has not changed.
*/
bool rekordbox_import_needed(void);
/*
* rekordbox_run_import() - full import pipeline.
*
* Caller must ensure database_tmp.tcd does NOT already exist (i.e. call
* this before do_tagcache_build / tagcache_build, or only when the temp
* file is absent).
*
* Steps performed:
* 1. Open /PIONEER/rekordbox/export.pdb
* 2. Parse artist/album/genre lookup tables
* 3. Parse track records
* 4. Write database_tmp.tcd in temp_file_entry format
* 5. Parse playlists and write .m3u8 files to /Playlists/Rekordbox/
* 6. Write /PIONEER/database.ignore
* 7. Update RB_SYNC_STAMP with the PDB's current mtime
*
* page_buf : scratch buffer for PDB page reads (must be ≥ 4096 bytes;
* pass a larger buffer if typical PDB page size is bigger —
* pdb_open() will report the required size and the function
* will bail if the buffer is too small).
* page_buf_size : size of page_buf in bytes.
* db_path : path to the Rockbox database directory (tc_stat.db_path).
*
* Returns number of tracks written to database_tmp.tcd, or -1 on error.
*/
int rekordbox_run_import(uint8_t *page_buf, uint32_t page_buf_size,
const char *db_path);
#endif /* HAVE_REKORDBOX */
#endif /* _REKORDBOX_IMPORT_H */

45
apps/tagcache.c
View file

@ -88,6 +88,9 @@
#include "dircache.h"
#include "errno.h"
#ifdef HAVE_REKORDBOX
#include "rekordbox_import.h"
#endif
#ifndef __PCTOOL__
#include "lang.h"
#include "eeprom_settings.h"
@ -5263,6 +5266,31 @@ static void tagcache_thread(void)
}
}
#ifdef HAVE_REKORDBOX
/* Rekordbox import: if export.pdb is present and newer than our last
* sync stamp, parse it into database_tmp.tcd before tagcache_build()
* runs. We use a 4 KB page scratch buffer allocated on the stack;
* pdb_open() will reject it if the actual page size is larger. */
if (!db_file_exists(TAGCACHE_FILE_TEMP) && rekordbox_import_needed())
{
static uint8_t rb_page_buf[4096];
logf("rekordbox: starting import");
int rb_result = rekordbox_run_import(rb_page_buf, sizeof(rb_page_buf),
tc_stat.db_path);
if (rb_result > 0)
{
/* Temp file is ready — commit it immediately. */
allocate_tempbuf();
commit();
free_tempbuf();
}
else if (rb_result < 0)
{
logf("rekordbox: import failed");
}
}
#endif /* HAVE_REKORDBOX */
#ifdef HAVE_TC_RAMCACHE
#ifdef HAVE_EEPROM_SETTINGS
if (firmware_settings.initialized && firmware_settings.disk_clean
@ -5359,6 +5387,23 @@ static void tagcache_thread(void)
logf("USB: TagCache");
usb_acknowledge(SYS_USB_CONNECTED_ACK, ev.data);
usb_wait_for_disconnect(&tagcache_queue);
#ifdef HAVE_REKORDBOX
/* Re-import if PDB changed during the USB session. */
if (!db_file_exists(TAGCACHE_FILE_TEMP)
&& rekordbox_import_needed())
{
static uint8_t rb_page_buf_usb[4096];
int rb_r = rekordbox_run_import(
rb_page_buf_usb, sizeof(rb_page_buf_usb),
tc_stat.db_path);
if (rb_r > 0)
{
allocate_tempbuf();
commit();
free_tempbuf();
}
}
#endif /* HAVE_REKORDBOX */
break ;
}
}

1
firmware/export/config/ipod6g.h
View file

@ -66,6 +66,7 @@
/* define this if you would like tagcache to build on this target */
#define HAVE_TAGCACHE
#define HAVE_REKORDBOX
/* define this if the unit uses a scrollwheel for navigation */
#define HAVE_SCROLLWHEEL

1
firmware/export/config/ipodcolor.h
View file

@ -48,6 +48,7 @@
/* define this if you would like tagcache to build on this target */
#define HAVE_TAGCACHE
#define HAVE_REKORDBOX
/* LCD dimensions */
#define LCD_WIDTH 220

1
firmware/export/config/ipodvideo.h
View file

@ -50,6 +50,7 @@
/* define this if you would like tagcache to build on this target */
#define HAVE_TAGCACHE
#define HAVE_REKORDBOX
/* LCD dimensions */
#define LCD_WIDTH 320