Discussion:
nosh version 1.14
Jonathan de Boyne Pollard
2015-05-09 23:09:35 UTC
Permalink
nosh is now up to version 1.14

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

These particular changelog entries are a big deal for Debian Linux.

* The previous regular sockets, sysinit services, and standard
targets packages are now all merged into the bundles package.
* There is now a new user VT package that runs new-style user VT
services.
* There is now a new kernel VT package that runs old-style kernel VT
services.
* The nosh-systemd-services package now automatically runs various
nosh subsystems under systemd.

I've taken the list of Debian packages available for download off the
infobox at the side of the blurb page and set up a page devoted
specifically to the Debian packages, explaining what they contain and
are for in more detail than could fit into that infobox.

*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/debian-binary-packages.html

There's a known bug that I'm working on. Because the preset conversion
is a little too aggressive, and because getttynam() exists on Debian
Linux, you'll currently need a dummy BSD /etc/ttys file that presets the
***@.service services that you want. Something like:

tty1 /bin/false linux on secure
tty5 /bin/false linux on secure
tty7 /bin/false linux off secure
vc0-tty /bin/false linux on secure
vc1-tty /bin/false linux on secure
vc2-tty /bin/false linux on secure

Those "vcN-tty" lines are user-mode virtual terminals. As you can
probably work out from this, on the BSD side the nosh user-mode virtual
terminal system is able to pull terminal login service enable/disable
information from the existing /etc/ttys configuration mechanism. (It
also can pull from /etc/rc.conf and does "onifconsole" too.)

The list of 157 things that I have to convert in order to fully replace
BSD /etc/rc.d is discussed in detail on the FreeBSD Hackers mailing
list. We have long since passed the point where it's possible to have
an entirely nosh-managed FreeBSD/PC-BSD system, though. The list of
things that I have to convert before I can likewise run my Debian Linux
system fully under the nosh system-manager is down to about a handful,
and is mainly the likes of service bundles for dbus and udev. If you
already have daemontools-style run scripts for those, or eudev, or mdev,
or whatever you enjoy, then you can race ahead of me. (-:

The page also warns about the nosh-bundles package potentially enabling
a lot of services. It's a package of over 400 service bundles. One way
of avoiding this is to go the only-enable-what-I-permit route, and use
this 99-default.preset:

disable *.service
disable *.socket

With something like this 00-administrator.preset alongside:

enable ***@acpid.service
enable ***@atd.service
enable ***@console-fb-realizer@*.service
enable ***@console-multiplexor@*.service
enable ***@gnucron.service
enable ***@kerneloops.service
enable ***@ModemManager.service
enable ***@NetworkManager.service
enable ***@org.cups.cups*.service
enable ***@polkitd.service
enable ***@terminal-emulator@*.service
enable ***@ttylogin@*.service
enable ***@update-binfmts.service
enable ***@wpa_supplicant.service
enable acpid.service
enable atd.service
enable console-fb-realizer@*.service
enable console-multiplexor@*.service
enable gnucron.service
enable kerneloops.service
enable ModemManager.service
enable NetworkManager.service
enable org.cups.cups*.service
enable polkitd.service
enable terminal-emulator@*.service
enable ttylogin@*.service
enable update-binfmts.service
enable wpa_supplicant.service

Adjust according to taste, of course. Mine also enables various
additional service bundles including dnscache, tinydns, http6d,
rabbitmq-server, and epmd (and their concomitant logging services) for
example.
--
To UNSUBSCRIBE, email to debian-user-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Archive: https://lists.debian.org/***@NTLWorld.com
Jonathan de Boyne Pollard
2015-06-01 00:02:56 UTC
Permalink
nosh is now up to version 1.16

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

As you'll see, the WWW pages have expanded a bit. In part this is
because of the Big News, which is the arrival of FreeBSD packages,
bringing FreeBSD up to par with Debian. The old box down the right-hand
side of the page was starting to make the thing look lop-sided. (-:

*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/freebsd-binary-packages.html
*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/debian-binary-packages.html

More big news on the package front is the reorganization into a main
"bundles" package and a group of "-run" packages. Lesser news is the
addition of packages for enabling/running various further groups of
services.

Comparatively small news are things like the change to the output of
"system-control status" and "service-status", which now uses long form
that displays more information. "svstat" retains its 1 line short form,
however. There's also a "system-control cat" command, for dumping out
service bundle configuration files. The new "emergency-login" fills the
gap where FreeBSD lacks a "sulogin" (because it's hardwired into the old
"init" and thus unusable separately), and also means that there's no
need to rely upon the old System 5 utilities/Linux utilities for
"sulogin" on Linux.

There is also a new roadmap WWW page. The Nosh Guide has also gained
several new pages dealing with logging and the import of external stuff.
--
To UNSUBSCRIBE, email to debian-user-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Archive: https://lists.debian.org/***@NTLWorld.com
Jonathan de Boyne Pollard
2015-06-28 15:19:42 UTC
Permalink
redo is now at version 1.2

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/redo.html

Two changes. First, as you can see, there's now a pre-built FreeBSD
binary package.

The second change is something that has been annoying me for some
while. Sometimes, especially when one is building packages, one ends up
with redo invoked by make. GNU make puts only flags in the MAKEFLAGS
environment variable. BSD make, however, puts at least one macro
definition in there too. redo was complaining about that, because it
was only expecting flags. Now it knows to ignore macro definitions. (-:

You'll need redo for one of the new (optional, of course) mechanisms in
nosh version 1.17.
--
To UNSUBSCRIBE, email to debian-user-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Archive: https://lists.debian.org/***@NTLWorld.com
Jonathan de Boyne Pollard
2015-06-28 16:39:04 UTC
Permalink
nosh is now up to version 1.17

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

As I said to gdiazhartusch a while back, and as listed on the roadmap
page, there is now a tool for auto-creating the mount@*, fsck@*,
ttylogin@*, dump@*, and swap@* service bundles from /etc/ttys and
/etc/fstab, and propagating settings from /etc/rc.conf{,.local} to
service bundle env/ directories. This is documented in the Nosh Guide,
in the section on external formats. It involves the use of redo, as I
just mentioned in the release annoucement for redo 1.2, because it's a
suite of redo scripts that endeavour to re-import the settings only when
the source files change. An important design choice, called out in the
introduction to the Nosh Guide, is that this is not a system of
continually-regenerated ephemera.

The Nosh Guide introduction has been completely rewritten, by the way.

The external formats import relies upon rcctl, which now has the full
implementation of "set" and "get" that was promised in the roadmap.
Note the caveat in the manual about not using "set" on a "status"
variable. That's an OpenBSD idiosyncrasy that wouldn't be correct for
FreeBSD even with the original tool. Use "enable" and "disable".
They'll work.

Part of importing /etc/fstab is generating appropriate dump@* and swap@*
service bundles. You'll note that swapauto and swaplate are no longer
services but targets. As services, they relied upon options to "swapon"
and were one giant lump. As targets, they pull in all of the generated
swap@* services which are individually enabled. Also note that whilst
the pre-built mount@-, fsck@-, mount@-usr, fsck@-usr, mount@-var, and
fsck@-var service bundles are still supplied, the auto-creation system
(or something similar) is meant to supersede them with bundles that
match your /etc/fstab after system installation.

Those aren't the only service bundle changes. There's the steady
chipping away at that list of 157 rc.d scripts, which has just lost off
its list a few pf services and savecore amongst other things. You might
be wondering why mountcritlocal and mountcritremote are still there.
The new auto-creation system makes mount@* and fsck@* bundles, and links
them into targets, after all. The answer is that those FreeBSD rc.d
scripts (like several others) have had some feature creep, and mounting
is not all that they in fact do. There's some temporary files cleanup in
there, for example.

Which leaves just the last major change, which is the /etc/
subdirectories. /etc/system-manager/targets and /etc/sv have been
brought together as /etc/service-bundles/targets and
/etc/service-bundles/services. And /etc/system-manager/presets is now
the better named /etc/system-control/presets as this is a function of
the latter tool not of the former. The post-upgrade scripts in the
binary packages will move the directories and create appropriate
symbolic links in the right places. If you are building from source and
hand-installing ... well look at what the post-upgrade scripts do, since
they are right there in the source package, and do that. (-:
--
To UNSUBSCRIBE, email to debian-user-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Archive: https://lists.debian.org/***@NTLWorld.com
Jonathan de Boyne Pollard
2015-08-20 13:01:22 UTC
Permalink
nosh is now up to version 1.18

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

The big news for this release is the nosh-run-system-manager Debian
binary package. This, and the new additional service bundles in
nosh-bundles, package up everything that is needed for running an
entirely nosh-managed basic Debian system with the nosh system-manager
program as process #1. And so the entry on the roadmap WWW page is
crossed out. Some notes:

* Don't forget that the Nosh Guide has a whole chapter on troubleshooting.
* With that package alone, you get very little running. This is
intentional. You'll have to install other nosh-run packages, or add
presets, for the various other things that you want. To get an OpenSSH
server running, for example, you'll need a local preset file (named,
say, /etc/system-control/presets/20-sshd.preset) with "enable sshd" and
"enable ***@sshd" before (re-)installing nosh-bundles.
(Re-)Installing the nosh-bundles package (re-)applies all current
presets, including your local ones, and auto-starts all enabled services.
* If you are running the freedesktop services, read the notes
hyperlinked-from the package download page.
* You may have spotted that there's a choice between running udev and
busybox mdev. (You pretty much must run one or the other for a fully
functional system.) The nosh-run-busybox-mdev package is broken. I
forgot to write the adapter tool. I've written it ready for version
1.19. There will be more said on the subject of busybox mdev in the
1.19 announcement, therefore.
* It's also intentional that you don't get System 5 shim commands for
the likes of "telinit" and "halt" unless you install the
nosh-systemv-shims package. "system-control poweroff" works without the
presence of the shims, of course.
* For novices, I recommend starting with nosh-run-kernel-vt .
nosh-run-user-vt still requires a manual step, after re-building the
service configuration each time, of "system-control disable
***@tty{1,2,3,4,5,6,7,8,9,10,11,12}".
* The recovery mode misbehaviour is a known problem. I'm
investigating. As a local fix, boot with init=/bin/sh on the kernel
command line and then run "exec /sbin/init -s" or even "exec /sbin/init
-b" from that shell prompt.

This is not the only news, of course. The BSD crowd should not feel
left out, moreover.

There are four long-standing problems with the Linux libkqueue library.
One of those problems causes svscan a.k.a. service-dt-scanner to be
spuriously woken up. This doesn't affect Debian but does affect Linux
operating systems such as Gentoo that have more recent versions of that
library. This has been worked around in version 1.18.

The pre-built mount@-, fsck@-, mount@-usr, fsck@-usr, mount@-var, and
fsck@-var service bundles have been removed. Generation of the service
bundles for mounting and checking volumes is now entirely based upon the
auto-creation system in /etc/system-control/convert/ . If you are
installing from scratch by hand, then you must remember to "redo all" in
that directory. The nosh-bundles package does this for you as part of
its post-install procedures.

The problem with the local-syslog-read service on Linux providing the
wrong socket (the BSD one) has been fixed.

The tools now speak true TAI, rather than UTC-10. There's an
explanation of the consequences of this in the manual pages for cyclog,
tai64n, and tai64nlocal.

The /etc/fstab conversion system now recognizes remote filesystem types
and attaches the generated services to remote-fs.target .
Jonathan de Boyne Pollard
2015-08-22 18:04:28 UTC
Permalink
nosh is now up to version 1.19

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

The important news is that the embarrassment with the post-install setup
script for the Linux nosh-run-kernel-vt package is fixed. It was a
missing 1-line escape() shell function. I apologize.

Other terminal management news is that there's now a console-clear
command that does pretty much the same thing as the Bourne Again shell's
clear_console command (also coming with that name as a symbolic link
alias) but better.

* The bash clear_console tries to open a lot of device files, as can be
seen in Ubuntu bug #39068. This tool by comparison doesn't need
anything more than its standard output, and doesn't attempt to open any
terminal devices itself at all.
* The bash tool is specific to the Linux kernel terminal emulator. It
had to be turned off for Debian kFreeBSD in Debian bug #355336, patched
to make it stop when run as the superuser in xterm in Debian bug
#355815, and worked around again in Debian bug #793883. This tool,
contrastingly, actually works with xterm and PuTTY and clears their own
scrollback buffers. It uses a different mechanism that both they and
(ironically) the Linux kernel terminal emulator since 2011, all support.
* Debian bug #791342 would be fixed by it, because it doesn't use the
bodge of attempting to switch virtual terminals away and back (using
either tty1 or tty2 as the "other" terminal) to clear the scrollback buffer.

On the gripping hand, this is something that one doesn't actually need
if one is using the nosh-run-user-vt package. console-terminal-emulator
supports the same extension to ECMA-48 Erase Display as xterm and PuTTY
do, but the raison d'être for clear_console is the likes of Debian bug
#331504. clear_console is in fact a ten-year-old bodge, addressing a
security/privacy concern that's a lot older still. With user-space
virtual terminals, one has the freedom to do things right, without the
need for such bodges. (-: As the console-terminal-emulator manual page
explains, when a login session terminates and the terminal is hung up,
the terminal emulator erases the whole display buffer.

In more other news: On Linux, fsck at bootstrap time is now monitored.
What this means from a user standpoint is that if your system reaches
its "maximum mounts before a forced full fsck" point, it doesn't just
sit there with nothing visibly happening (if one cannot see the hard
disc activity light) for ages. The fsck@* services now invoke
"monitored-fsck" rather than fsck directly. This is an ordinary
chain-loading tool that opens a client connection to a local (i.e.
AF_LOCAL) socket and then chains to fsck adding in its (Linux-specific)
-Cfd option. There's a new monitor-fsck-progress service that runs the
server for that socket, and displays progress information on the
console. This latter is intentionally replaceable by alternative
services, of course. I'm intending to make its output somewhat
prettier, rather than just dumping the raw information at you as it does
in this release. But if you want to write your own ...

You'll have to delete /etc/system-control/convert/volumes (or modify the
contents of /etc/fstab) and run "redo all" to get your existing
auto-created fsck@* service bundles regenerated with the new command.
Or just edit the run files replacing fsck with monitored-fsck .

The big news is that as promised in the 1.18 announcement the
nosh-run-busybox-mdev package is now functional. Also as promised in
that announcement, here's more on the subject.

The nosh toolset doesn't come with a bunch of rules for your
plug-and-play manager, be that (BSD) devd, (Linux) udev, or busybox
mdev. Your plug-and-play manager does, or should do. As packaged up
for Debian Linux, udev comes with a whole bunch of pre-supplied rules in
/lib/udev/rules/ that gets one the "usual" device file tree in /dev/ .
And it almost goes without saying that the BSDs come with devd rules in
the box. The same is not true for the busybox Debian package. There's
no /etc/mdev.conf supplied.

You MUST write one before using busybox mdev. busybox mdev's default
behaviour as packaged, in the absence of /etc/mdev.conf , may be logical
and straightforward; but it does not result in a working Debian system.
Some things that I've hit myself are /dev/null not being accessible to
anyone except the superuser, which affects loads of things all over the
shop, and event device files not being under /dev/input/ where other
parts of the system expect them to be.

There's plenty to read on this subject in the non-Debian world, starting
with but not limited to:

* https://wiki.gentoo.org/wiki/Mdev
* http://linuxfromscratch.org/clfs/view/clfs-3.0/mips/bootscripts/mdev.html

You'll have to adapt these for Debian. There are also the examples in
/usr/share/doc/busybox/examples/ , of course, the larger of the two
fixing both of the aforementioned problems. The positive news is that
the busybox-mdev service implicitly serializes invocations of mdev, so
that there's no need for mucking around with mdev's sequence number
mechanism.

The recovery mode problem mentioned in the 1.18 announcement turns out
to have a simple local fix, which I'll incorporate into a more
comprehensive service fix:

# ln -s rescue /etc/service-bundles/targets/single
Jonathan de Boyne Pollard
2015-09-28 01:05:44 UTC
Permalink
The nosh package is now up to version 1.20 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

It's worth noting that the WWW site has gained some more pages, an
installation how-to and a quick look at user-space virtual terminals.

*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/timorous-admin-installation-how-to.html
*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/user-vt-screenshots.html

The command and tool list page, which was woefully out of date, has had
some attention, too. It is rather longer than it was.

*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/commands.html

You might notice a couple of new BSD packages, as well. FreeBSD/PC-BSD
binary packaging is now up to parity with Debian Linux. One can create
a fully-nosh-managed system on both just by installing some binary packages.

This wipes another to-do item off the roadmap page. The list of
remaining rc.d items on the roadmap has shrunk, also. As always,
assistance in wiping those remaining rc.d items off the list is
welcome. If someone feels up to tackling /etc/rc.d/bluetooth, perhaps
looking at what Iain Hibbert has apparently already done, for example ...

In addition to having yet more service bundles, this release irons out
some wrinkles in startup and shutdown. The sysinit phase of bootstrap
was causing undesirable mounts in emergency mode. That has been
restructured. Some ordering problems in shutdown relating to unmounting
filesystems have also been fixed. And the System 5/BSD compatibility
reboot, halt, and poweroff shims no longer rely upon some other
toolset's (not necessarily even present) shutdown command.

There are now -run packages for four different Debian Linux
plug-and-play managers, with vdev and suckless mdev now added.
Joe Maloney
2015-09-28 01:33:42 UTC
Permalink
Hi,
do you have a source code repository somewhere for nosh? Like on GitHub?

Joe Maloney


On Sun, Sep 27, 2015 at 8:05 PM, Jonathan de Boyne Pollard <
Post by Jonathan de Boyne Pollard
The nosh package is now up to version 1.20 .
*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
It's worth noting that the WWW site has gained some more pages, an
installation how-to and a quick look at user-space virtual terminals.
*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/timorous-admin-installation-how-to.html
*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/user-vt-screenshots.html
The command and tool list page, which was woefully out of date, has had
some attention, too. It is rather longer than it was.
*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/commands.html
You might notice a couple of new BSD packages, as well. FreeBSD/PC-BSD
binary packaging is now up to parity with Debian Linux. One can create a
fully-nosh-managed system on both just by installing some binary packages.
This wipes another to-do item off the roadmap page. The list of remaining
rc.d items on the roadmap has shrunk, also. As always, assistance in
wiping those remaining rc.d items off the list is welcome. If someone
feels up to tackling /etc/rc.d/bluetooth, perhaps looking at what Iain
Hibbert has apparently already done, for example ...
In addition to having yet more service bundles, this release irons out
some wrinkles in startup and shutdown. The sysinit phase of bootstrap was
causing undesirable mounts in emergency mode. That has been restructured.
Some ordering problems in shutdown relating to unmounting filesystems have
also been fixed. And the System 5/BSD compatibility reboot, halt, and
poweroff shims no longer rely upon some other toolset's (not necessarily
even present) shutdown command.
There are now -run packages for four different Debian Linux plug-and-play
managers, with vdev and suckless mdev now added.
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
Jonathan de Boyne Pollard
2015-09-28 07:27:43 UTC
Permalink
do you have a source code repository somewhere for nosh?
*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/source-package.html

The source package and how to build from source are here.
Jonathan de Boyne Pollard
2015-10-05 05:39:29 UTC
Permalink
The nosh package is now up to version 1.21 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

console-terminal-emulator now has a mouse input event protocol, and
speaks both the DEC VT Locator protocol and the xterm Private Mode 1006
protocol over the terminal interface to applications. These are the
protocols that you get with ttymouse=dec and ttymouse=sgr in vim. Since
support for the 1006 protocol is fairly widespread in the relevant
places nowadays, it seemed not worthwhile implementing the inferior
Private Mode 1005 and Private Mode 1015 protocols. Moreover,
console-terminal-emulator is UTF-8 and the Private Mode 1005 protocol
has ambiguities once one introduces UTF-8.

console-fb-realizer as yet only talks to real mouse input devices on
Linux, but handling FreeBSD/PC-BSD mouse input devices is on the
roadmap. This has necessitated a change in the command-line syntax of
console-fb-realizer, and concomitant changes in the pre-supplied
realizer service bundle, which will need to be updated in tandem if you
are using user-space virtual terminals. This change allows the mouse
input device to be specified in addition to the keyboard input event
device. It also slightly regularizes display-only mode, which is
signified now by the simple lack of any mouse or keyboard device
specifications, rather than by an explicit option. Yes, I am aware that
there's no mouse cursor sprite drawn yet. Acutely so.

FreeBSD improvements include the completion of geli and gbde import that
the sharp-eyed might have noticed quietly part-done in version 1.20.
/etc/fstab entries for volumes using these should now be translated into
appropriate interdependent mount@*, gbde@*, and geli@* service bundles.
This is rather difficult for me to test, though, as noted on the roadmap.

The big PC-BSD improvement is jails support, which has lurked at the
bottom of the roadmap page for a while. Both PC-BSD Warden and FreeBSD
9 jails are now recognized by the external configuration import
subsystem, and converted into appropriate service bundles. The mechanism
here is fairly straightforward: The jails themselves are one service
bundle, and the programs that run in the jails are another. The latter
service is after/ and wants/ the former service. The jexec command is a
chain loading tool that modifies process state in the same vein as
setuidgid, softlimit, and envdir, and one can simply employ it as such.
The jail command can be used analogously, with jail -c and jail -r, to
how the mount and umount commands in mount@* service bundles are used.
Those are what the import subsystem does.

Importing Warden Linux jails isn't available yet; and some of the more
esoteric FreeBSD 9 rc.conf and PC-BSD Warden METADIR/* jail options are
not yet imported. Enabling jails to be autostarted at bootstrap is via
the "jails" and "warden" targets, by the way.

The rc.d conversion project has progressed, with a few more things wiped
off. As mentioned in the version 1.20 message, all assistance in wiping
the final 40-odd FreeBSD rc.d scripts off the list, to be found on the
roadmap page, is welcome. And if any PC-BSD people have ideas on how to
turn things like /usr/local/etc/rc.d/pc-samba into service bundles,
those are welcome too. (Note that pre-supplied service bundles already
exist for the Linux flavours of some of these, which may or may not be a
starting point.)
Jonathan de Boyne Pollard
2015-11-02 02:15:55 UTC
Permalink
The nosh package is now up to version 1.22 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

There are several things in this release:

* a new binary package for FreeBSD
* improvements to the user-mode virtual terminal subsystem
* changes and additions to UCSPI tools
* log export to remote servers
* kqueue on Linux
* miscellany

a new binary package for FreeBSD
--------------------------------

There's now a debian-shims binary package for FreeBSD. This contains the
heretofore not packaged invoke-rc.d and update-rc.d shim programs. I
decided to make this separate from the systemv-shims package because
these areless general-purpose than those shims are.

The haltsys, fasthalt, fastpoweroff, and fastboot shims are now
packaged, also.

improvements to the user-mode virtual terminal subsystem
--------------------------------------------------------

The console-fb-realizer now displays a mouse pointer sprite on the
display, to reflect the position of the mouse, when an application has
turned it on with the relevant control sequences. Mouse support via the
evdev input subsystem on Linux is thus now fully implemented, including
support for tablets that use absolute rather than relative positioning.

On the FreeBSD side, you can use sysmouse devices. But this only
permits relative positioning. This is a limitation of sysmouse itself,
as far as I can see. A lot has to change, including the kernel, the
protocol, and moused, to enable absolute positioned devices via
sysmouse. Absolute positioning devices will therefore be supported using
uhid devices. Some of that is done already, but it's not complete yet.

Keyboard maps are now generated by the external configuration import
subsystem from whatever one has in /usr/share/vt/keymaps , rather than
being hardwired to a fixed set of countries. In the absence of this
directory (as will usually be the case on Linux operating systems),
fallback U.S.A. and U.K. keyboard maps are generated.

This generation is worthy of note, as it exemplifies the mechanism that
allows multiple BSD keyboard maps to be overlaid to make one generated
map. The fallback U.K. keyboard map is generated by taking the built-in
U.S.A. keyboard map and applying a "us_to_uk" overlay map on top of it
that only has the few differences between the U.S.A. International
layout and the U.K. one. (This currently produces the basic U.K.
layout. "U.K. Extended" should be a simple matter of another overlay
that does the various Option+A -> a-acute mappings, but that's
somethingfor the future.) Similarly, versions of existing maps that
swap Caps Lock and Control are generated by adding a simple overlay that
does solely that. Likewise, generated maps have an overlay applied that
sets the Backspace key to the application-programmable DEC VT behaviour
that console-terminal-emulator supports, that out-of-the-box BSD keymaps
don't know anything about.

changes and additions to UCSPI tools
------------------------------------

For consistency, the UCSPI tools that supported a single --numeric
option now support --numeric-host and --numeric-port options, for
separately determining whether hosts and ports are taken to be names or
just numbers.

There are now client-side tcp-socket-connect and udp-socket-connect
tools, that open client sockets, connect them to servers, and then
chain. These adhere to the UCSPI conventions for inherited open file
descriptors in client-side tools.

log export to remote servers
----------------------------

The new UCSPI clients were motivated by the new export-to-rsyslog
command. This is a daemon that expects to be invoked as a UCSPI client,
connected to a remote RFC 5424/RFC 5426 ("rsyslog") server. It
maintains a set of "log cursors" that point to daemontools-stylelog
directories. Tracking its position in the logs using those cursors, it
sends new log information to the connected server. In the usual nosh
fashion, the filesystem is the database, and the "cursors" are just
files and symbolic links. The details are on the manual page.

In conjunction with the UCSPI clients, export-to-rsyslog thus makes a
log remote export service.

This isn't intended to be the last word in such things. RFC 5426is
unreliable, and RFC 5424 loses the microsecond and nanosecond
information of TAI64N. But it demonstrates the idea and shows that this
can be done in the daemontools world. One can indeed export
daemontools-stylelogsif one has (say) a suite of servers whose log data
should be copied over, on the fly, to a centralized rsyslog server.
There's room here for someone to take this idea and run further with it
using something like RELP.

miscellany
----------

The several miscellaneous items include OpenLDAP services in the
autoconfiguration subsystem and some tweaks to the /etc/fstab conversion
on Linux to deal with records that don't explicitly say either
read-write or read-only, resulting in undocumented behaviour in the
Linux fstab parsing library.

On the subject of working around the behaviours of Linux libraries ...

libkqueue
---------

Those familiar with the development will know that Linux's libkqueue has
been a perennial difficulty. Its inaccessible private internal file
descriptors are not marked close-on-exec, leaving open a security hole
if libkqueue is used in a privileged process that forks off unprivileged
children to run other programs. NOTE_WRITE for EVFILT_VNODE isn't
implemented correctly for directories. And itbreaks when inotify events
come through that have filenames in them. Those familiar with the code
will know that there was quite a lot of conditional compilation as a
consequence, replacing libkqueue with individual hand-rolled mechanisms
in those programs where libkqueue simply doesn't work or creates
security weaknesses. The final straw was a user reporting
service-dt-scanner abending on Gentoo Linux when the scan directory is
merely listed with "ls", which we eventually tracked down to libkqueue bugs.

No more.

I tried the route of patching libkqueue. It was my preferred route.
It's fairly easy to see where to add in the close-on-exec flags, for
example. The difficulty is in getting such things available both to
users using Debian binary packages (on all of its various
"stable"/"oldstable" flavours) and to users building from source on
distributions that I don't have myself. In the end I took a step back
and pondered whether libkqueue was even the right thing to be using in
the first place. After all, it's built to select from a multiplicity of
implementations for several operating systems, using an internal
abstraction layer,where the nosh toolset is only in fact ever using it
for one.

So there's now an internal C++ kqueue/kevent library for Linux in the
toolset, not ideal but "good enough" for the use that the nosh toolset
needs from kqueue/kevent and doing the various things that it needs like
close-on-exec, inotify with filenames and multiple events in one go, and
proper NOTE_WRITE for directories; and the conditional code, the
individual hand-rolled mechanisms (apart from one), and the binary
package dependenciesfrom libkqueue are now gone.
Jonathan de Boyne Pollard
2015-12-17 13:50:04 UTC
Permalink
The nosh package is now up to version 1.23 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

There is one major item in this release.

* I've adjusted console-fb-realizer's keyboard handling on BSD to use
USB directly.

There are a few more minor changes.

* I've upgraded the version of clang++ that is used to build the binary
packages to 3.8.0. This should have no visible effect whatsoever. (-:
* The handling of the DECDA2 control sequence by
console-terminal-emulator now copes with what vim sends. (What vim
sends isn't what my DEC VT tests had been checking.)
* convert-systemd-units now inserts uses of the ionice and chrt
chain-loading commands on Linux into the generated service bundles.

Mostly this is a clearing the decks release in the hope that I will be
able to do some more work on the remaining few FreeBSD conversions
before the new year.

USB keyboard support
====================

The keyboard handling is a change to using the USB HID devices
(/dev/uhid*) on FreeBSD in preference to (but not forcibly instead of)
the ATA keyboard protocol. In part this is in order to handle the
"consumer" keys that USB has. In part this is in order to handle the
extra keys that one finds on 106-key, 107-key, and 109-key keyboards and
on some numeric keypads (such as the ABNT2 thousands-separator key). In
part it's to remove an extra layer of the user-space virtual terminal
system that can be outwith the kernel. In part it's to match the USB
mouse capability from version 1.22 of the toolset.

Please note that the structure of kbdmap files has changed slightly, to
accomodate mappings for "consumer" keys, to reposition the entries for
some of the 106/107/109-key keyboards' extra keys, and to cover all of
the function key gymnastics that vim can accept. The
/etc/system-control/convert/ system should automatically re-convert your
VT kbd files into the new format.As part of this, I've moved the mapping
for the Euro symbol in the fallback U.K. layout (as generated on Linux
in the absence of VT kbd files). It used to be level 3 shift on the
[eE] key in prior versions of the toolset. Almost all real U.K.
keyboards nowadays have it engraved as level 3 shift on the [4$] key,
and that's where it now is.

Also note that I'm still working on this. There might be further
changes in 1.24. I've found a U.K. keyboard with two [#~] keys (at A00
and C12), and I need to check out whether this actually employs what I
had thought to be an error in the USB HID usage tables (distinct usages
for "\|" and "Europe1") and had corrected, or whether this is a quite
mad keyboard that simply has two "Europe1" keys (or two "\|" keys).
Also, I've ordered an ABNT2 and a Japanese USB keyboard, and hope to do
some testing with them, which may prompt further tweaks. (I really
wanted to buy a Leadership 4530 keyboard. They seem to be out of stock
in a lot of places.)
Jonathan de Boyne Pollard
2016-01-13 09:25:07 UTC
Permalink
The nosh package is now up to version 1.24 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

Minor items in this release include:

* A fix for BSD keyboard layout import, that makes both "duml" and
"ddia" be U+0308 for now. Technically, diaeresis and umlaut are
distinguishable in Unicode decomposed forms (using U+034F). But for
now, everything is simple unadorned combining diaeresis.
* A few more service bundles, for DBMail and for sudo (which in its
vanilla form puts its timestamp files in /var/lib instead of /var/run
and needs a cleanup service -- see Debian Bug #786555).
* Use of rtprio and idprio when converting system service units on
FreeBSD/PC-BSD.
* Improvements to the framebuffer video mode selection in user-space
virtual terminals for FreeBSD/PC-BSD. It now comes up in the same
display size as on Debian Linux on my test machines.
* Doco and other fixes from user feedback on version 1.23. (I've
already begun some further VirtualBox host adjustments, as we discussed,
for 1.25.)

There is one major item in this release.

PC-BSD 10.2
===========

Until now, I'd been testing on a PC-BSD system that had been upgraded,
with various contortions, from version 9. This was still using UFS
filesystems, listed in /etc/fstab; which the external configuration
import subsystem had been happily importing to native service bundles.
Over Christmastide I installed a PC-BSD 10.2 system from scratch,
discovering some interesting oddities. These included installation
failing if you tell it that you are in the United Kingdom using a U.K.
keyboard (PC-BSD Bug #12986); and the GRUB menu editor, as configured by
the installer, operating in a truly eye-stretching 46 column by 28 row
mode (by my count), and not displaying the underscore character correctly.

The important thing to know is that PC-BSD has for some time (at least
since 2013) been ZFS-only, as far as installation goes. (One can of
course mount other filesystem types after installation.) As Henry Ford
might have said "Any customer can install to any filesystem type that xe
likes, as long as it is ZFS.". The result is that if installing from
scratch one gets a whole load of ZFS datasets, and an empty (save for
/proc and swap) /etc/fstab file.

So the major push for version 1.24 has been to get the configuration
import system to deal with this, which it now does. It will create
mount services for all ZFS mounts, enable the ones that are "on", give
them an inter-service ordering, and deal with the special-casing for the
root (which the installer, oddly, marks as not automatically mounted,
even though it of course is).

Alongside this, a large chunk of the remaining NetBSD rc.d services,
from the on-going project to entirely replace them, have been crossed
off the list. These include mfs for /tmp, static networking and static
ARP, pefs, serial port BPS and framing setup, ppp, rfcomm_pppd,
persistent "entropy" for the randomness subsystem, and ipfw. The
progress of this work has been open from the start, and you can follow
along on the roadmap WWW page. Indeed, you can even join in, if you can
convert any of the remaining few items.

There's more work to be done. But I now have ZFS-only PC-BSD 10.2
running nosh system-managed and service-managed.

Some notes for those eager to follow:

* Yes; I'm working on a pcdm service. No; it doesn't help that it's
undocumented. Yes; that hoopla and palaver with forked subshells and
multiple while loops calling sleep is exactly the sort of thing that
proper service management is intended to obviate.

* If you have problems with devd, stale nologin from previous boots, and
other things that use /var/run, it's because the convert_varrun service
isn't enabled and your system has not been thus or otherwise migrated to
/run. This will be properly enabled by a preset in the next version.
Enable it and reboot. Or just start it and reboot. Or just boot into
rescue mode and turn /var/run into a symbolic link to /run yourself.

* No; the nosh-run-system-manager package doesn't work properly on
PC-BSD, as it does on vanilla FreeBSD. PC-BSD 10.2 doesn't use the
FreeBSD boot loader, like my old upgraded installation of PC-BSD 9 did.
It uses GRUB. The PC-BSD people apparently plan to get rid of GRUB in
the future, and use the FreeBSD loader once more. So this problem goes
away when GRUB does. (-: In the meantime, use 'set
kFreeBSD.init_path="/sbin/service-manager"' in the GRUB configuration.

* The root-resizing subsystem that was new to FreeBSD version 10 still
needs conversion. But ironically it doesn't work on PC-BSD 10.2 in the
first place. It can only grow UFS volumes, and PC-BSD's root is not a
UFS volume.
Jonathan de Boyne Pollard
2016-02-01 01:03:17 UTC
Permalink
The nosh package is now up to version 1.25 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

As you may have noticed from discussions elsewhere, a new
oom-kill-protect utility has snuck in at the last moment. This takes
Linux-style OOM Killer score adjustments (an integer between -1000 and
1000), BSD-style binary YES/NO settings, or a special setting for
querying the "oomprotect" environment variable; and tries to do the
closest matching thing for each platform. Details are in the manual, of
course. With this, the OOMScoreAdjust setting is now converted by the
convert-systemd-units utility.

The local-syslogd, udp-syslogd, and syslogd service bundles make use of
oom-kill-protect with the special environment variable setting in their
run programs. So FreeBSD bug #204741 is addressed in a more general
fashion that can be easily used in other service bundles. "rcctl set
syslogd oomprotect YES" and "rcctl set syslogd oomprotect NO" can be
used to turn OOM Killer protection on and off.

Other things in this version include:

* More configuration import utilities, covering ip6addrctl, webcamd, and
NFS settings.
* A fix for a problem with configuration import on Linux in version 1.24.
* Two minor utilities for querying the fstab database, get-mount-what
and get-mount-where, needed by the configuration import for mdconfig
(but generally usable).
* New binary "run-" packages for OpenSSH server, syslog on a local
socket, and klog.
* The new syslog and klog packages provide the Debian package manager's
virtual package names "linux-kernel-log-daemon" and "system-log-daemon"
(per Debian Bug #67604).

As can be seen from the roadmap, we are nearing the end of the rc.d
conversion for FreeBSD. Additions in this release include nfsserver,
gptboot, rtadvd, virecover, and pcdm. Almost all of mdconfig is
actually done, bar some after/before orderings.

*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/roadmap.html#FreeBSDrc.d
Jonathan de Boyne Pollard
2016-05-06 23:19:09 UTC
Permalink
The nosh package is now up to version 1.27 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

In fact, it is soon to be version 1.28. This is a somewhat delayed
notice for 1.27, because I forgot to send out the notices for versions
1.27 and 1.26 after updating the WWW site.

As can be seen from the roadmap, we are at the point in rc.d conversion
for FreeBSD/PC-BSD where it's actually easier to count the things that
remain unconverted. Discounting the PC-BSD Active Directory services
and a handful of suspect FreeBSD services (such as growfs, which doesn't
apply to ZFS in the first place) the remaining things to be converted
can be counted on the fingers of one hand. The external configuration
import mechanism has gained the ability to handle stf, atabridge,
mdconfig, and a few others. There are also a whole bunch more service
bundles: cross-platform, for Linux, and for BSD.

*
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/roadmap.html#FreeBSDrc.d

The OOM Killer avoidance measures from version 1.25 are now employed in
the PostgreSQL service bundle as well.

The systemd service unit conversion tool has gained a whole load of
NUMA-related extensions: NUMAInterleave, NUMAMemBind, NUMACPUNodeBind,
NUMAPhysCPUBind, NUMALocalAlloc, and NUMAPreferred. These it translates
into the equivalent invocations of the numactl chain-loading utility.

It has also gained a couple of minor fixes and tweaks. The %m
substitution now works, and service bundles comprising FIFOs or AF_LOCAL
sockets are now created so that they are ordered after any relevant
filesystem mount services.

By request, the nosh Guide has gained a whole chapter of cheatsheets,
giving quick one-liner pointers to some common tasks. The chapter is
divided into three sections: chain loading, logging, and service
management. The service management division is subdivided into
daemontools-style commands, systemd-style commands, OpenBSD-style
commands, SMF-style commands, and common commands. The chain loading
division gives a number of the more common commands used in
chain-loading run scripts (and whereever else one might want to use them).

There have been improvements in static network setup, including fixes
for some bugs in static_arp and static_ndp and a more cross-platform
replacement for the static-networking service.

The nosh-bundles package now supplies several aliases for services,
which are just plain old symbolic links. So (for example) one can
address the CUPS service as either org.cups.cupsd or just plain cupsd.

Things to look forward to in version 1.28 already include: more service
bundles; another chain-loading utility; a major revision to MySQL and
MariaDB service bundling, to reflect the pushes by their own developers
to obviate their rc scripts and the mysql-safe command and just run
mysqld directly under service management using the tools provided by the
service management system; and a change relevant to the all-important
linux_logo command. (-:
Jonathan de Boyne Pollard
2016-08-06 17:30:19 UTC
Permalink
The nosh package is now up to version 1.28 .

* https://jdebp.eu./Softwares/nosh/
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

There's a lot in this one: MySQL and MariaDB changes; more prophylaxis
for Desktop Bus bus activation; improvements to systemd unit conversion;
support for the old svc -x; machineenv; improvements to service
management; fixes for the per-user manager; improvements to the console
terminal emulator; BSD boot mode changes; the ability to pass more open
sockets to connection-accepting programs; cron; and OpenBSD.

Italics and colour
==================

* https://jdebp.eu./Softwares/nosh/italics-in-manuals.html

This isn't a toolset change, per se. But the WWW site now has a guide
to seeing actual italic text in manual pages. The nosh toolset's
user-space virtual terminals support true italics (if one has the fonts)
or obliquing, and this works with them.

MySQL and MariaDB changes
=========================

* https://jdebp.eu./Softwares/nosh/mariadb-and-mysql.html

New in version 1.28 is a different and up-to-date way of managing MySQL
and MariaDB server services — where "new" translates to finally getting
rid of that unnecessary mysql_safe wrapper and doing things the way that
daemontools-family toolset users have wanted to do them since the turn
of the century.

There's a lengthy exposition on the WWW site, q.v.. The major visible
effect is that your "mysql" or "mariadb" service is now an alias, for
something like a "mysql@" or "***@01" (if you have [mysql01] in your
my.cnf) service. The configuration file import mechanism tries to
construct/update ***@NN and ***@NN service bundles for you, based
upon your MariaDB and MySQL configuration files.

Further prophylaxis for Desktop Bus bus activation
==================================================

* https://jdebp.eu./Softwares/nosh/avoid-dbus-bus-activation.html

The nosh toolset now comes with a dbus-daemon-launch-helper
replacement. The purpose of this is to sit in your
/usr/local/etc/dbus-1/system.conf (or equivalent) and redirect to
service management attempts, by the Desktop Bus broker daemon, to
demand-start services. It is slightly fiddly to install, requiring
manual setup by the system administrator, there being no simple way to
add overrides to /usr/local/etc/dbus-1/system.conf and it requiring that
you allow the "messagebus" user the necessary access for starting and
stopping services (but not necessarily *superuser* access — rembember ACLs).

To assist with this, several popular Desktop Bus "services" now exist as
alias names for service management services. These are just symbolic
links to the service bundle directories, of course. So, for example:
With the helper in place, Desktop Bus bus activation will try to
demand-start a service named "org.freedesktop.PackageKit" using service
management. This is just an alias for the "packagekit" service.

Improvements to systemd unit conversion
=======================================

Ideal mode is now closer to the daemontools-family mainstream,
defaulting to the daemontools-family norm of always restarting
services. Quirks mode, conversely, now implements more of the
non-daemontools redirection semantics for standard I/O, in particular
with regard to listening socket units. Some more Linuxisms have been
added. Limits (where applicable) can now take SI and IEC suffixes (so
you can, say, express limits in kiloseconds). This latter is actually
an augmentation to the underlying softlimit command.

Passing more open sockets to connection-accepting programs
==========================================================

The improvements to systemd unit conversion also allow passing more than
one listen()ing socket to connection-accepting programs. You can use,
say, ListenStream and ListenDatagram and the conversion utility will
translate this into an appropriate chain of multiple invocations of
udp-socket-listen and tcp-socket-listen. It will do
local-stream-socket-listen, local-datagram-socket-listen,
netlink-datagram-socket-listen, and fifo-listen too.

The motivator for this was Daniel J. Bernstein's dnscache. I have
modified versions of tinydns, dnscache, and taiclockd that understand
the LISTEN_FDS protocol for their being told about listening sockets
that have been opened for them, and don't open their own sockets in that
case. dnscache, in particular, takes a UDP socket and a listening TCP
socket. The UCSPI tools in this version of the toolset can now provide
these two to a dnscache process. One simply chains through
udp-socket-listen and tcp-socket-listen to dnscache, using the
--systemd-compatibility flag.

The sharp-eyed will notice that the tinydns and dnscache services are
following in the footsteps of the mariadb and mysql services, being
instantiated for relevant IP addresses by the configuration import
subsystem instead of being single-instance services. It's not quite
ready in this version, but you can see where things are headed.

svc -x
======

The -x option to service-control, a.k.a. svc, is no longer defunct. It
does what unload-when-stopped does.

machineenv
==========

This was a little helper that was needed for running one particular Java
program under nosh service management. It's analogous to userenv, but
instead of setting up environment variables from login information it
sets up environment variables from machine information.

Improvements to service management
===================================

The system-control utility now respects a new "use_hangup" flag in
service directories, as documented in service-manager's manual. This is
used in the various getty@, agetty@, mgetty@, emergency-login@, and
ttylogin@ service bundles, and causes system-control to attempt to bring
down these services (at shutdown, for example) with SIGHUP, because some
job control shells ignore SIGTERM and that delays shutdown.

Incidentally: There's now a family of pre-supplied agetty/mgetty/getty@
service bundles, set up for the various conventional serial terminal
devices on OpenBSD, FreeBSD/PC-BSD, and Linux. These are aliased from
parallel ttylogin@ names, for consistent handling of /etc/ttys import
and so forth. They run a modem-aware getty, though; which of course the
virtual terminal ttylogin@ services do not. See the Terminals chapter
of the nosh Guide.

Fixes for the per-user manager
==============================

The per-user manager, an instance of which you get when you run
"system-control start user@${USER}" (or, more specifically,
"system-control start user-services@${USER}"), is now more functional.
The configuration importer sets you up with your own personal service
bundle directory tree in ${HOME}/.config/service-bundles/ with a
pre-supplied exit/ service bundle. (Don't delete it! The per-user
service manager needs to have a goal when it is signalled to terminate.)

Improvements to the console terminal emulator
=============================================

SoftTerm has a bugfix. CUF and CUF no longer cause sideways scrolling.
It also now understands some of the more esoteric control sequences from
the old (pre-version 10) FreeBSD kernel terminal emulator, increasing
the compatibility with programs that use the "cons25" terminal type.

The NetBSD terminal type for virtual terminals is now pcvtXX, and the
OpenBSD one is pccon, per the 2015 termcap database.

There is also a minor fix in the SCO Console mode emulation. All of the
function key control sequences were off by one. They are not, now. (-:

BSD boot mode changes
=====================

* https://jdebp.eu./FGA/emergency-and-rescue-mode-bootstrap.html

I'd like to see FreeBSD and PC-BSD kernel loaders support the -b flag.
In the meantime, for simple utility, I've changed the -s flag to invoke
emergency mode rather than rescue mode, but only on FreeBSD/PC-BSD.

OpenBSD
=======

One of the servers providing the new WWW site is an OpenBSD machine,
running the aforementioned modified versions of Bernstein tinydns and
dnscache, and Bernstein publicfile, under nosh service management. The
nosh toolset now builds and runs on OpenBSD 5.9.

There are an awful lot of limitations to OpenBSD, from lack of realtime
signals and nmount() to its older packaging tool. Some of them are
surmountable: I could write an nmount() shim function as I did for
Linux, although for OpenBSD it would be a lot more complex. Some of
them, like the lack of fexecve(), are not. In concert, they preclude
nosh as a system manager and packages with full automatic setup and
teardown of basic system services.

But one can run the nosh service manager under OpenBSD rc; and
everything else, from UCSPI servers to cron, under that. I currently do.

I'm not particularly bothered about the package tool and the missing
setup/teardown of the binary packages, myself. I'm not running my
system using installed binary packages in the first place. I'm building
from source, in true slashpackage style, into /package/admin/*/command/
and symbolically linking from /usr/local/bin/* to there.

The user-space virtual terminal system has not been tested on OpenBSD,
and almost certainly doesn't quite work yet. As mentioned earlier, the
OpenBSD terminal type is set correctly by vc-get-tty. But
console-terminal-emulator does not have an OpenBSD mode yet;
console-fb-realizer doesn't really know how to use OpenBSD HID devices
yet; and OpenBSD has the old 1980s-style pseudo-terminal management
system instead of the "new" "UNIX98" 1990s one, which I have not
tested. At one point I was in a halfway-done position of having the
nosh toolset's UCSPI tools but none of the service bundles. So the
toolset now has a tcpserver shim, that simply maps onto
tcp-socket-listen, ucspi-socket-rules-check, and tcp-socket-accept. It
does not have the same defaults as the Bernstein tcpserver, though; and
is in a separate binary package.

The cron mess
=============

OpenBSD has yet another not-quite-the-same version of cron, with yet
another way to tell it not to uselessly fork. So now there's an
openbsd-cron service bundle alongside the debian-cron and vixiecron
service bundles. The cron alias is set up to point to the right one for
the operating system.

debian-cron is the new name for the gnucron service bundle, by the way.
"Debian/GNU Linux's prepackaged version" is what Ian Jackson originally
called it in its README. (Historical tidbit: Paul Vixie originally
named his "PD cron".) As far as I can tell, GNU cron was actually a
project (by one Mike Meyer it appears) for the Free Software Foundation
in 1987 that has entirely disappeared, if it ever existed in the first
place. In any case, "gnucron" is too generic a name for something that
is at this point heading down the road of needing an individual service
bundle for every single special flavour. (Perhaps the Google Summer of
Code people could add importing OpenBSD's cron's -n flag to FreeBSD cron
to their list of ideas. That would eliminate one special flavour and
fix a deficiency at the same time.)

Thibault Godouet's fcron, Matt Dillon's/Jim Pryor's dcron, and Bruce
Guenter's bcron don't need special treatment like this.
Julian Elischer
2016-08-14 14:10:06 UTC
Permalink
On 7/08/2016 1:30 AM, Jonathan de Boyne Pollard wrote:

I don't know if I just missed it, or it isn't there but I have a
question..
You give examples of importing systemd service files. What about
importing rc.d files with all their ability to run arbitrary shell
commands.
And once you have the services defined, what is the logical equivalent
of rc.conf, which can supply parameters for each service and turn them
on and off? can you import from rc.conf?
Jonathan de Boyne Pollard
2016-08-21 11:04:22 UTC
Permalink
I don't know why you asked about FreeBSD rc.d just on the Debian mailing
list; but I'm going to deal in both of those and others besides, here,
and things that apply across both, so I've re-included the FreeBSD
Post by Julian Elischer
I don't know if I just missed it, or it isn't there but I have a
question..
You give examples of importing systemd service files. What about
importing rc.d files with all their ability to run arbitrary shell
commands.
And once you have the services defined, what is the logical equivalent
of rc.conf, which can supply parameters for each service and turn them
on and off? can you import from rc.conf?
You did miss it. (-:

What you missed has grown to be a significant subsystem. It was actually
mentioned a couple of times in the 1.28 announcement. It's the external
configuration import subsystem. You can read about it in the nosh Guide:

xdg-open /usr/local/share/doc/nosh/external-formats.html

As you can see, there's a whole section on importing from rc.conf into
native service management mechanisms. ("rc.conf" covers several
sources, note, including a FreeNAS configuration database and
/etc/defaults/rc.conf .)

The native service mangement mechanisms are the "enable" and "disable"
subcommands to the system-control command, and using the envdir command
in the normal daemontools-family style way. The enable/disable
mechanism in "rc.conf" is treated as if it were a preset (in systemd
nomenclature). You tell service management to "preset" a service, and
it will look at /etc/rc.conf and /etc/rc.conf.local (as well as some
other preset mechanisms) to determine what to set the native
enable/disable state to. The user manual page for the preset subcommand
(of system-control) explains what the preset mechanisms are in detail.

You can set up environment directories how and where you like, but
there's a convention that is shared by the "convert-systemd-units" tool,
the "rcctl" shim, and the external configuration import subsystem as a
whole. This convention is an environment directory named "env" that is
in the service directory. The "rcctl" shim gets and sets variables
there; and the import subsystem places converted "rc.conf", /etc/fstab,
/etc/ttys, /etc/my.cnf, and other stuff there.

One example of this in action, out of many in the import subsystem, is
jails that have been set up the version 9 way in "rc.conf". Those are
turned into service bundles, with "env" environment directories that
contain environment settings such as "hostname", "mount_devfs", and
"interface". The "run" script for the jail service very simply turns
the environment variables into arguments to the "jail" comand. In a
system with an original OpenBSD "rcctl" command, one would expect to be
able to set (version 9) jail control variables by manipulating
/etc/rc.conf with commands like "rcctl set wibble hostname wobble". The
"rcctl" shim and this shared convention mean that one need not stray
that far from this if "rcctl" is one's habit: "rcctl set v9-***@wibble
hostname wobble" does the "native" thing of setting the "hostname"
variable in the (conventional) environment variable directory for the
"v9-***@wibble" service.

Bonus feature for those with other habits: With nosh service management
in place, one can actually import from /etc/rc.conf settings *on Debian*
(as long as one sets up a FreeBSD/PC-BSD-style /etc/defaults/rc.conf
pointing to it with rc_conf_files). One can use /etc/ttys, too.

As for importing scripts that run "arbitrary shell commands", there are
several points.

First, you may not need to. Note that most of what you get out of the
box in /etc/rc.d/ and /usr/local/etc/rc.d/ on FreeBSD and PC-BSD has
already been converted. Remember that project that I had to convert 157
services? Take a look at the nosh roadmap page. It's almost done.

Second, you may not need to. Take a look at what actually comes in the
nosh-bundles package nowadays. Discounting the 'cyclog@' service
bundles there are just over 540 service bundles in there, from samba to
ntp, from saned to ***@agentd. (Including the 'cyclog@' service
bundles, it is over a thousand service bundles.) The Debian world
doesn't get left out, either. Although it's a lot more difficult than in
the BSD worlds to come up with a list of "core" Debian services, a lot
of the basics of Debian are also covered by this, from kernel-vt-setfont
through irqbalance to update-binfmts. And those more-than-540 service
bundles cover lots of "non-core" stuff, from (as aforementioned)
OSSEC-HIDS, Salt, and RabbitMQ to publicfile httpd over IPV6.

Third, you may not need to. This was mentioned in the 1.28
announcement, in fact. The external configuration import subsystem
makes *further* service bundles, beyond the pre-made ones that come in a
binary package. It creates service bundles to run (optional) per-user
service management, per-user Desktop Bus brokers, MySQL and MariaDB
servers (according to your my.cnf), PPP and SPPP, md and pefs, jails
(set up with v9 rc.conf or the PC-BSD Warden), tinydns and dnscache
services (not quite ready when 1.28 came out, as the announcement said),
static IP4/IP6/ARP/NDP setup and teardown, and more besides.

Fourth, you may not need to. Out of all of this, there's probably
already an existing service bundle for something similar that one can
copy and adapt.

Fifth, you may not need to. The convert-systemd-units tool exists,
after all. If there's a system service/socket unit around, converting
that may well be simpler starting point than starting with an rc
script. It's usually significantly simpler than starting with a van
Smoorenburg rc script, although Mewburn and OpenBSD rc scripts can
themselves be fairly simple starting points. I did a "make fetch" on
the PC-BSD ports tree a couple of months ago. (As an aside: there are
several broken ports that don't do the right thing here.) There are a
growing number of packages where there's now a systemd service/socket
unit in the fetched source archive.

Sixth, the easy cases are easy. As just noted, Mewburn and OpenBSD rc
scripts can themselves be fairly simple. (They are not *always* so,
though, contrary to popular belief.) If you have an rc script that says
"The command name is this, its arguments are that.", it is very easy
indeed to convert this into something that can execute as a "run"
program. Setting up all of the stuff around the "run" program for a
complete service bundle is merely an exercise in two-line shell scripts
(for things like "start" and "stop") and making directories and symbolic
links (for things like the "before" and "wants" directories).

Seventh, the hard cases require a human being anyway. Parsing a shell
script that runs "arbitrary shell commands" would require creating what
is essentially a full shell script interpreter, that can handle the
Almquist, Bourne Again, and Korn shell syntaxes (because such a
hypothetical *general-case* conversion tool would have to address van
Smoorenburg rc scripts on Debian, Mewburn rc scripts on
FreeBSD/PC-BSD/NetBSD and friends, and OpenBSD rc scripts) that knows
about at least five quite different sets of "helper" commands (from
start-stop-daemon to startproc) and that works out how an entire shell
script translates into the actual acts of executing one or (in really
bad cases) more services. At this point, I defer to a human being
*understanding what is needed* and writing one or more service bundles. (-:

And there is, of course, scads of doco, written over the past two
decades by many people, on how to write daemontools-family-style "run"
scripts.
Joe Nosay
2016-08-30 15:58:27 UTC
Permalink
Thank you very much.
And have a blessed day.

On Sun, Aug 21, 2016 at 7:04 AM, Jonathan de Boyne Pollard <
Post by Jonathan de Boyne Pollard
I don't know why you asked about FreeBSD rc.d just on the Debian mailing
list; but I'm going to deal in both of those and others besides, here, and
things that apply across both, so I've re-included the FreeBSD mailing
I don't know if I just missed it, or it isn't there but I have a
Post by Julian Elischer
question..
You give examples of importing systemd service files. What about
importing rc.d files with all their ability to run arbitrary shell commands.
And once you have the services defined, what is the logical equivalent of
rc.conf, which can supply parameters for each service and turn them on and
off? can you import from rc.conf?
What you missed has grown to be a significant subsystem. It was actually
mentioned a couple of times in the 1.28 announcement. It's the external
xdg-open /usr/local/share/doc/nosh/external-formats.html
As you can see, there's a whole section on importing from rc.conf into
native service management mechanisms. ("rc.conf" covers several sources,
note, including a FreeNAS configuration database and /etc/defaults/rc.conf
.)
The native service mangement mechanisms are the "enable" and "disable"
subcommands to the system-control command, and using the envdir command in
the normal daemontools-family style way. The enable/disable mechanism in
"rc.conf" is treated as if it were a preset (in systemd nomenclature). You
tell service management to "preset" a service, and it will look at
/etc/rc.conf and /etc/rc.conf.local (as well as some other preset
mechanisms) to determine what to set the native enable/disable state to.
The user manual page for the preset subcommand (of system-control) explains
what the preset mechanisms are in detail.
You can set up environment directories how and where you like, but there's
a convention that is shared by the "convert-systemd-units" tool, the
"rcctl" shim, and the external configuration import subsystem as a whole.
This convention is an environment directory named "env" that is in the
service directory. The "rcctl" shim gets and sets variables there; and the
import subsystem places converted "rc.conf", /etc/fstab, /etc/ttys,
/etc/my.cnf, and other stuff there.
One example of this in action, out of many in the import subsystem, is
jails that have been set up the version 9 way in "rc.conf". Those are
turned into service bundles, with "env" environment directories that
contain environment settings such as "hostname", "mount_devfs", and
"interface". The "run" script for the jail service very simply turns the
environment variables into arguments to the "jail" comand. In a system
with an original OpenBSD "rcctl" command, one would expect to be able to
set (version 9) jail control variables by manipulating /etc/rc.conf with
commands like "rcctl set wibble hostname wobble". The "rcctl" shim and
this shared convention mean that one need not stray that far from this if
the "native" thing of setting the "hostname" variable in the (conventional)
Bonus feature for those with other habits: With nosh service management in
place, one can actually import from /etc/rc.conf settings *on Debian* (as
long as one sets up a FreeBSD/PC-BSD-style /etc/defaults/rc.conf pointing
to it with rc_conf_files). One can use /etc/ttys, too.
As for importing scripts that run "arbitrary shell commands", there are
several points.
First, you may not need to. Note that most of what you get out of the box
in /etc/rc.d/ and /usr/local/etc/rc.d/ on FreeBSD and PC-BSD has already
been converted. Remember that project that I had to convert 157 services?
Take a look at the nosh roadmap page. It's almost done.
Second, you may not need to. Take a look at what actually comes in the
there are just over 540 service bundles in there, from samba to ntp, from
over a thousand service bundles.) The Debian world doesn't get left out,
either. Although it's a lot more difficult than in the BSD worlds to come
up with a list of "core" Debian services, a lot of the basics of Debian are
also covered by this, from kernel-vt-setfont through irqbalance to
update-binfmts. And those more-than-540 service bundles cover lots of
"non-core" stuff, from (as aforementioned) OSSEC-HIDS, Salt, and RabbitMQ
to publicfile httpd over IPV6.
Third, you may not need to. This was mentioned in the 1.28 announcement,
in fact. The external configuration import subsystem makes *further*
service bundles, beyond the pre-made ones that come in a binary package.
It creates service bundles to run (optional) per-user service management,
per-user Desktop Bus brokers, MySQL and MariaDB servers (according to your
my.cnf), PPP and SPPP, md and pefs, jails (set up with v9 rc.conf or the
PC-BSD Warden), tinydns and dnscache services (not quite ready when 1.28
came out, as the announcement said), static IP4/IP6/ARP/NDP setup and
teardown, and more besides.
Fourth, you may not need to. Out of all of this, there's probably already
an existing service bundle for something similar that one can copy and
adapt.
Fifth, you may not need to. The convert-systemd-units tool exists, after
all. If there's a system service/socket unit around, converting that may
well be simpler starting point than starting with an rc script. It's
usually significantly simpler than starting with a van Smoorenburg rc
script, although Mewburn and OpenBSD rc scripts can themselves be fairly
simple starting points. I did a "make fetch" on the PC-BSD ports tree a
couple of months ago. (As an aside: there are several broken ports that
don't do the right thing here.) There are a growing number of packages
where there's now a systemd service/socket unit in the fetched source
archive.
Sixth, the easy cases are easy. As just noted, Mewburn and OpenBSD rc
scripts can themselves be fairly simple. (They are not *always* so,
though, contrary to popular belief.) If you have an rc script that says
"The command name is this, its arguments are that.", it is very easy indeed
to convert this into something that can execute as a "run" program.
Setting up all of the stuff around the "run" program for a complete service
bundle is merely an exercise in two-line shell scripts (for things like
"start" and "stop") and making directories and symbolic links (for things
like the "before" and "wants" directories).
Seventh, the hard cases require a human being anyway. Parsing a shell
script that runs "arbitrary shell commands" would require creating what is
essentially a full shell script interpreter, that can handle the Almquist,
Bourne Again, and Korn shell syntaxes (because such a hypothetical
*general-case* conversion tool would have to address van Smoorenburg rc
scripts on Debian, Mewburn rc scripts on FreeBSD/PC-BSD/NetBSD and friends,
and OpenBSD rc scripts) that knows about at least five quite different sets
of "helper" commands (from start-stop-daemon to startproc) and that works
out how an entire shell script translates into the actual acts of executing
one or (in really bad cases) more services. At this point, I defer to a
human being *understanding what is needed* and writing one or more service
And there is, of course, scads of doco, written over the past two decades
by many people, on how to write daemontools-family-style "run" scripts.
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
Jonathan de Boyne Pollard
2016-12-03 11:33:27 UTC
Permalink
The nosh package is now up to version 1.29. *
http://jdebp.eu./Softwares/nosh/ *
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/ There's been a lot going on since
version 1.28 . 2016 leap second ---------------- The TAI to UTC
conversions know about the forthcoming leap second. service bundles
--------------- As usual, there are several new service bundles, from
powerd++ through zfsd to fwknopd. The new fs-servers target allows one
to order the initialization of NFS servers before loop-to-self NFS
mounts. The new multi-user-pre target is another ordering target that
allows services such as the motd file updater to be ordered before TTY
login services. The instantiated ***@tty7 and ***@ttyv6 services have
been replaced with a single kdm service, with a view to dealing with
display managers better in the future. I have some plans in this area.
The Samba service names have been fixed. Debian calls them nmb, smb, and
winbind; but the Samba doco and most places on the WWW call them nmbd,
smbd, and winbindd. The latter names are used in the service bundles
package, with aliases pointing to them from the Debian names. doco ----
The doco has been improved and kept up-to-date in various places,
including correct descriptions of set-service-env and print-service-env
after one confused user got in touch. PC-BSD is now named as TrueOS
where the reference is not historical. code review ----------- As a
result of some code review that was offered, std::auto_ptr is now gone
and a rare memory corruption bug in safe_execvp() has been fixed.
Building from scratch when one doesn't have a prior daemontools or
freedt toolset installed also no longer hits a bug. configuration import
improvements --------------------------------- In an effort to clear
those last few remaining items on the nosh roadmap, a whole load of
configuration import (pppd, sppp, rfcomm_ppp, dhclient, wpa_supplicant,
natd, and hostapd) has been consolidated under the umbrella of
static-networking. I plan to expand this further in 1.31, given how much
is already in 1.30. Linux kernel VTs ---------------- Management of
Linux kernel virtual terminals has some improvements, including setting
UTF-8 canonical mode editing and keyboard composition modes, and
emitting the control sequences that set up the screen saver. tai64nlocal
changes ------------------- tai64nlocal has adopted a minor but
important change from the BSD and GNU C libraries: before reading the
start of a line it flushes its output. This came from trying to use it
as a co-process in GNU awk. To prevent deadlocks, GNU awk co-processes
need to be in what is effectively line buffered output mode even though
their standard inputs and outputs are not terminal devices. This is now
the case for tai64nlocal and it can be used to convert TAI64N timestamps
as a GNU awk co-process. FreeBSD and TrueOS packaging
---------------------------- The largest change, however, is in the
FreeBSD/TrueOS and OpenBSD packaging. This is a change that is going to
happen in the Debian packaging in a later version. It's partly to
simplify the package maintenance, and partly a step towards having
OpenBSD packages that work. A single package description is fed to both
the new pkg tool that exists on FreeBSD/TrueOS and the old pkg tool that
exists on OpenBSD. It's not perfect, as there are things that are easy
with the new pkg tool that are hard with the old one; and the OpenBSD
packages are still not fully functional. But things are better than they
were. The OpenBSD service bundles package now almost properly sets up
per-service user accounts and log directories, for example.
=======================================================================
=========== IMPORTANT UPGRADE NOTE FOR FreeBSD/TrueOS: ===============
=======================================================================
An important consequence of the aforementioned is that the semantics of
the nosh-bundles package have changed. In earlier versions, the various
nosh-run-* packages were how one set services running, except for a
small rump set of services that were set up by the nosh-bundles package.
This is now no longer the case. The nosh-bundles package now presets and
starts no services at all. *All* running of services must be achieved
with the nosh-run-* packages or some other sets of scripts and presets.
To this end, there are now two new packages,
nosh-run-freebsd-desktop-base and nosh-run-freebsd-server-base. These
parallel the already existing nosh-run-trueos-desktop-base and
nosh-run-trueos-server-base packages; except that they do not start any
of the services that exist in TrueOS but do not exist in FreeBSD, such
as the various pc-* services. You must install, for a working
fully-nosh-managed system, exactly one of these four packages. If you
are running nosh service management under Mewburn rc, you can of course
run as many or as few services under the nosh service manager as you
care to switch over from Mewburn rc. But if you are running a
fully-nosh-managed system these packages will arrange to run the various
fundamentals that one pretty much cannot do without, such as
mounting/unmounting volumes, running devd and ldconfig, and initializing
the PRNG.
Jonathan de Boyne Pollard
2016-12-03 11:41:04 UTC
Permalink
Bloody Thunderbird! Here's that again, I hope without the surprise
reformatting after pressing "send" this time:

The nosh package is now up to version 1.29.

* http://jdebp.eu./Softwares/nosh/
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

There's been a lot going on since version 1.28 .

2016 leap second
----------------

The TAI to UTC conversions know about the forthcoming leap second.

service bundles
---------------

As usual, there are several new service bundles, from powerd++ through
zfsd to fwknopd. The new fs-servers target allows one to order the
initialization of NFS servers before loop-to-self NFS mounts. The new
multi-user-pre target is another ordering target that allows services
such as the motd file updater to be ordered before TTY login services.
The instantiated ***@tty7 and ***@ttyv6 services have been replaced with
a single kdm service, with a view to dealing with display managers
better in the future. I have some plans in this area.

The Samba service names have been fixed. Debian calls them nmb, smb,
and winbind; but the Samba doco and most places on the WWW call them
nmbd, smbd, and winbindd. The latter names are used in the service
bundles package, with aliases pointing to them from the Debian names.

doco
----

The doco has been improved and kept up-to-date in various places,
including correct descriptions of set-service-env and print-service-env
after one confused user got in touch. PC-BSD is now named as TrueOS
where the reference is not historical.

code review
-----------

As a result of some code review that was offered, std::auto_ptr is now
gone and a rare memory corruption bug in safe_execvp() has been fixed.
Building from scratch when one doesn't have a prior daemontools or
freedt toolset installed also no longer hits a bug.

configuration import improvements
---------------------------------

In an effort to clear those last few remaining items on the nosh
roadmap, a whole load of configuration import (pppd, sppp, rfcomm_ppp,
dhclient, wpa_supplicant, natd, and hostapd) has been consolidated under
the umbrella of static-networking. I plan to expand this further in
1.31, given how much is already in 1.30.

Linux kernel VTs
----------------

Management of Linux kernel virtual terminals has some improvements,
including setting UTF-8 canonical mode editing and keyboard composition
modes, and emitting the control sequences that set up the screen saver.

tai64nlocal changes
-------------------

tai64nlocal has adopted a minor but important change from the BSD and
GNU C libraries: before reading the start of a line it flushes its
output. This came from trying to use it as a co-process in GNU awk. To
prevent deadlocks, GNU awk co-processes need to be in what is
effectively line buffered output mode even though their standard inputs
and outputs are not terminal devices. This is now the case for
tai64nlocal and it can be used to convert TAI64N timestamps as a GNU awk
co-process.

FreeBSD and TrueOS packaging
----------------------------

The largest change, however, is in the FreeBSD/TrueOS and OpenBSD packaging.

This is a change that is going to happen in the Debian packaging in a
later version. It's partly to simplify the package maintenance, and
partly a step towards having OpenBSD packages that work. A single
package description is fed to both the new pkg tool that exists on
FreeBSD/TrueOS and the old pkg tool that exists on OpenBSD. It's not
perfect, as there are things that are easy with the new pkg tool that
are hard with the old one; and the OpenBSD packages are still not fully
functional. But things are better than they were. The OpenBSD service
bundles package now almost properly sets up per-service user accounts
and log directories, for example.

=======================================================================
=========== IMPORTANT UPGRADE NOTE FOR FreeBSD/TrueOS: ===============
=======================================================================

An important consequence of the aforementioned is that the semantics of
the nosh-bundles package have changed. In earlier versions, the various
nosh-run-* packages were how one set services running, except for a
small rump set of services that were set up by the nosh-bundles
package. This is now no longer the case. The nosh-bundles package now
presets and starts no services at all. *All* running of services must be
achieved with the nosh-run-* packages or some other sets of scripts and
presets.

To this end, there are now two new packages,
nosh-run-freebsd-desktop-base and nosh-run-freebsd-server-base. These
parallel the already existing nosh-run-trueos-desktop-base and
nosh-run-trueos-server-base packages; except that they do not start any
of the services that exist in TrueOS but do not exist in FreeBSD, such
as the various pc-* services.

You must install, for a working fully-nosh-managed system, exactly one
of these four packages. If you are running nosh service management
under Mewburn rc, you can of course run as many or as few services under
the nosh service manager as you care to switch over from Mewburn rc.
But if you are running a fully-nosh-managed system these packages will
arrange to run the various fundamentals that one pretty much cannot do
without, such as mounting/unmounting volumes, running devd and ldconfig,
and initializing the PRNG.
Jonathan de Boyne Pollard
2016-12-31 23:53:24 UTC
Permalink
The nosh package is now up to version 1.30 .

* http://jdebp.eu./Softwares/nosh/
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

service bundles
---------------

As usual, there are more service bundles, including for the UWSGI
"Emperor" and the new services in FreeBSD/TrueOS 11 such as ypldap.
There are now services to run Sendmail in the same manner as the
services that run exim. Note that this is slightly different to the old
FreeBSD division of labour. There are individually controllable
services for SMTP Submission, SMTP Relay, the Submission queue runner,
and the Relay queue runner.

doco
----

The Guide has been extended with several new chapters, including a
gazetteer of interesting directories, a chapter on log file
post-processing, a chapter on logging security, a chapter on per-user
service management, and some notes for individual services. The
commands list has moved from the blurb into the Guide, too, as it seems
like something that an administrator might find handy to have available
when there's no Internet connection.

* http://jdebp.eu./Softwares/nosh/guide.html

service management
------------------

There's now a hardlimit chain-loading command, analogous to softlimit.
The convert-systemd-services utility now makes use of this and permits
setting separate hard and soft limits, or only one or the other, with
settings like LimitOFILE=32:128 and LimitNPROC=:infinity .

There's now a local-reaper chain-loading command, that can turn "local
reaper" status for the current process on or off. Have a care when
using this, per the note on the manual page. There is a
LocalReaper=true extension to systemd service units for this.

netlink-datagram-socket-listen is now available on the BSDs for script
compatibility. It always aborts with an address family error.

There's a new hangup subcommand of system-control, equivalent to the
existing -H option to svc .

enhancements to system-control stop/start/reset and single-shot services
------------------------------------------------------------------------

This is the first big item for 1.30 :

The start and stop subcommands of system-control now operate more
quickly. Instead of polling once per second, they monitor the
supervise/status files of each service that is in the process of being
started and stopped, with kevent().

In addition, system-control now supports the notion of services that
become ready when their main process has exited, marked with a new flag
file in the service directory. convert-systemd-units has been modified
to convert "oneshot" services to this, instead of to services that put
all of the run code into the start program. Thus "oneshot" services
that are running their actual main programs are reported as "running" by
svstat, rather than as "starting".

This takes advantage of the extended status information that
service-manager has been writing to the status file since version 1.28.
The sharp-eyed may have noticed that in version 1.28 the output of
"svstat"/"system-control status" gained information about the exit
statuses of the start, run, restart, and stop programs. This is what
system-control now uses to detect whether ready-after-run services ran
before they stopped. (Detection of ready-after-run services that are
running with no processes, because they are "remain" services, can be
and is done with just the daemontools-encore-compatible status information.)

Old-style "oneshot"s will continue to work as before, as of course they
become ready as soon as the run process is spawned, which is after they
have run their programs as part of start.

The benefit of this new style, apart from reporting a running service as
actually "running", which should help with nagios monitoring and the
like, is that "oneshot" services converted from systemd no longer have
to be marked as RemainAfterExit=true in order to avoid a dummy "pause"
process hanging around. This is the case for old-style "oneshot"
services. They have to run something in run, after all, and that
something has to keep running in order for the service to be considered
ready and services ordered after it to be unblocked. A ready-after-run
service, however, unblocks ordered-after services if it has reached the
stopped state via a run, thus puts its programs in run, thus doesn't
have to have a dummy pause process, and can be RemainAfterExit=false
without adding to the process list.

log file management
-------------------

export-to-rsyslog had a bug that caused it to skip old log files (the
@nnnnnnnnnnnnnnnnnnnnnnnn.s ones) in catch-up mode. This has been
corrected. There is now a follow-log-directories command that can
substitute for tail -F . It knows the actual structure of log
directories, operates using one or more cursors like export-to-rsyslog
does, and copes correctly with cyclog/multilog log rotation (which GNU
tail, at least, apparently has problems with when the timing is
particularly wrong on a loaded system).

See also http://jdebp.eu./FGA/do-not-use-logrotate.html

build
-----

More warnings are now turned on with clang++ during the build, and a lot
of the resultant warnings have been eliminated where appropriate. The
check for eg++ in preference to g++ is now limited to OpenBSD, where (at
least on OpenBSD 5.9) eg++ is still ahead of g++ by a wide margin.

Per-user service management
---------------------------

Changes in per-user service management are the second big item for 1.30 :

The per-user service manager instances are now invoked via userenv, so
all per-user services that you run under nosh service management, D-BUS
servers or otherwise, will have your own HOME, SHELL, and USER set.
Several per-user daemon softwares were expecting HOME to be set.

To match what the Desktop Bus people are doing, the dbus socket path for
the per-user D-BUS broker has changed from
"/run/user/$USER/dbus/user_bus_socket" to "/run/user/$USER/bus". In
theory, this is addressable (in D-BUS speak) as "unix:runtime=yes". In
practice, there is no version of D-BUS available on stable/release
FreeBSD, TrueOS, or Debian that understands this address syntax. So one
still has to use "unix:path=/run/user/$USER/bus".

The Desktop Bus people and the desktop environments people are also
switching from per-login D-BUS brokers to per-user D-BUS brokers. The
nosh toolset has already had this for over a year, since the middle of
2015. Each real-person user account has an optional per-user service
management service (e.g. user-***@fred). What is new is that
per-user service bundle areas are now populated with a whole load of
service bundles for real services, many relating to GUI desktop
environments, and the per-user D-BUS broker has moved to there, from
being a system-level service bundle.

The configuration import subsystem creates these new per-user service
bundles in the home directories of individual real users, under
~fred/.config/service-bundles/services/ and
~fred/.config/service-bundles/targets/ (for user fred). These run
per-user services for a whole load of things, from GNOME editor and
emacs through dconf and KDE Notify to urxvtd and GNOME Terminal.

The configuration import subsystem also sets up a bypass for D-BUS's
broken "bus activation" mechanism, so that instead of attempting to run
these D-BUS servers directly, the D-BUS broker instead tells the nosh
per-user service manager to run them. This takes the form of a
replacement dbus-daemon-launch-helper, and the per-user D-BUS brokers
now employ a modified configuration file that invokes it.

There's a full explanation of how this all works in the new chapter on
demand-starting user-level Desktop Bus services in the nosh Guide.

Notes:
* For emacs as a per-user service, you must have a very recent emacs
with its very-late-to-the-party --new-daemon option.
* GNOME Weather and its interaction with GeoClue2 are only partly
tested, because the versions of them available for the test platforms
were attempting to contact a weather service that the U.S. Government
discontinued in June 2016; and this was hardwired into their code.
Jonathan de Boyne Pollard
2017-01-14 11:26:10 UTC
Permalink
The nosh package is now up to version 1.31 .

* http://jdebp.eu./Softwares/nosh/

*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

* http://jdebp.info./Softwares/nosh/

This release fixes a problem with emergency mode that was introduced by
accident in 1.29 . The emergency-***@console service was not properly
enabled by package installation. Now it once again is.

There are a number of bug fixes in this release, such as rare corner
cases in how convert-systemd-units generates arguments to pass to sh,
what port the nginx server part of Appcafe binds to when not the
default, the use of setuidgid-fromenv to set more than 1 supplementary
group ID, and making the Makefile in tinydns@* services work with both
BSD and GNU make. Various service bundles that perform
clean-up-directories actions at bootstrap have been made more difficult
to accidentally re-trigger after bootstrap.

There is also a fair amount of new features:

* The automatically-generated data for tinydns@* services now
encompasses all of the reverse lookup domain names for private/local IP
addresses, so none of the DNS traffic involving such lookups will leak
out of your machine/organization to the rest of Internet.

* The userenv command has gained the ability to (optionally) set a whole
lot more environment variables from the capabilities in /etc/login.conf
and ~/.login_conf . It now can be used as the
setup-the-user-environment part of a command chain that is designed to
perform the setup of an interactive login session. This is particularly
useful for fixing PCDM, the display manager in TrueOS.

* The pipe command can now arrange to clean up the child process in one
of two ways. This is made use of in the dnscache service bundles, and
dnscache services no longer contain the perpetual zombie process that
they had in version 1.30 .

* Presets now support wildmat-style character set wildcards. e.g. one
can now write "***@vc[0-9]-tty" as a service name pattern.

* If you have been using the --verbose option to the start/stop/reset
subcommands of system-control, you'll notice that it now colourizes its
output. Its output has also been adjusted to more clearly indicate
blocked services and what they are blocked by.

The big item is that there is now a complete set of simple control
groups manipulation commands, the pre-supplied service bundles all make
use of it, and all service bundles created by convert-systemd-units make
use of it. (All of this is a no-op on FreeBSD/TrueOS and OpenBSD, of
course.)

If you've read the Linux doco, you'll know that control groups do not
require any sort of centralized gatekeeper process, and are a
decentralized system that can be driven with just the echo command. In
practice, using echo is non-trivial. The move-to-control-group,
delegate-control-group-to, and set-control-group-knob commands take the
hassle out of working out exactly what to echo where. They do all of
the hard work of determining what the directory name of the current
control group under /sys/fs/cgroup is, and present a simple system
allowing one to create and navigate to another control group, delegate
control over the current control group (and its subgroups) to an
unprivileged user, and set control group knobs.

The set-control-group-knob utility further illustrates the convenience
functionality over and above a simple echo command. It can calculate a
knob setting as a percentage of another number, handle SI and IEEE/IEC
multiplier suffixes, and translate the device file names that are
(comparatively) convenient for humans into the literal major and minor
device numbers that the Linux control groups API actually operates in
terms of.

There are new chapters in the Guide covering the automatic import of
FreeBSD 9 and PC-BSD Warden jails, how jailing services on
FreeBSD/TrueOS works, and limiting services. The limiting services
chapter covers both the original Unix resource limits system and Linux
control groups.
Jonathan de Boyne Pollard
2017-01-30 09:09:24 UTC
Permalink
The nosh package is now up to version 1.32 .

* http://jdebp.eu./Softwares/nosh/

*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

* http://jdebp.info./Softwares/nosh/

This release fixes two problems with Gentoo Linux (control group version
detection and a problem with mounting API filesystems) that we hashed
out on the Supervision mailing list. It furthermore contains a change
to the way that convert-systemd-units generates service bundles that
fixes problems with control group setup when the service unit defines a
"slice" for the service or when the service unit is a template. In
furtherance of that there's a new create-control-group command.

Other things in this release include improvements to the (unpackaged) Z
Shell command-line completions, which now display option completion
menus properly; some improvements to the Terminals chapter in the Guide;
fixes to various service bundles that were using shell reserved words
and operators such as "for" and "&&" without explicitly invoking the
shell; additions to userenv for setting DBus and XDG Runtime variables;
and a fix that prevents "system-control reset" from looping indefinitely
when run by an unprivileged user such as "messagebus" that lacks access
to the control/status API.

The major improvement in this release, though, is to console-fb-realizer
on TrueOS.

FreeBSD gives console-fb-realizer uhid device files to use for input
devices, which speak the USB HID report protocol and which
console-fb-realizer has been happy with for a long time. TrueOS
provides either ums/ukbd devices, which lack various features because
they speak the old sysmouse and atkbd protocols, or ugen devices. There
are no uhid devices available. console-fb-realizer can now use the ugen
devices. Moreover, it will detach the ums/ukbd drivers from the ugen
devices using the new detach-kernel-usb-driver command, so that there
aren't two things both attempting to read HID reports.

console-fb-realizer also now correctly sets the keyboard LEDs on both
FreeBSD and TrueOS.

There have been several minor adjustments to the kernel VT sharing parts
of console-fb-realizer, preparatory to splitting the program up into
separate parts for input and output devices, permitting things such as
multiple keyboards each with its own keyboard map and numlock semantics,
in a future release.
Jonathan de Boyne Pollard
2017-04-09 19:52:07 UTC
Permalink
The nosh package is now up to version 1.33 .

* http://jdebp.eu./Softwares/nosh/
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

This has been held back because of work being done by someone else. I don't
want to steal xyr thunder, so I'll leave the announcement of that work to xem.
Suffice it to say that it will interest a new group of people.

There are several major improvements in 1.33 .

Packaging
---------

In the version 1.29 announcement I said that the Debian packaging system was
going to be brought into line with the system used for FreeBSD/TrueOS and
OpenBSD. This is now done. Debian and the BSDs all now use a similar system
for generating each package manager's package maintenance instructions from an
abstract package description.

==============================================================
=========== IMPORTANT UPGRADE NOTE FOR Debian: ===============
==============================================================

An important consequence of the aforementioned is that the semantics of the
nosh-bundles package have changed. In earlier versions, the various nosh-run-*
packages were how one set services running, except for a small rump set of
services that were set up by the nosh-bundles package.

This is now no longer the case. The nosh-bundles package now presets and starts
no services at all. *All* running of services must be achieved with the
nosh-run-* packages or some other sets of scripts and presets.

To this end, there are now two new packages, nosh-run-debian-desktop-base and
nosh-run-debian-server-base. These parallel the
nosh-run-{freebsd,trueos}-{desktop,server}-base packages already available since
1.29 for FreeBSD/TrueOS. You must install, for a working fully-nosh-managed
system, exactly one of the nosh-run-debian-{desktop,server}-base packages.

If you are running nosh service management under systemd, you can of course run
as many or as few services under the nosh service manager as you care to switch
over from systemd. But if you are running a fully-nosh-managed system these
packages will arrange to run the various fundamentals that one pretty much
cannot do without, such as mounting/unmounting volumes, running
udev/eudev/vdev/mdev, binfmt loading, and initializing the PRNG.

Log service account names
-------------------------

The naming scheme used for the user accounts for dedicated log service users has
changed. Installing the new nosh-bundles package should automatically rename
all existing log service accounts to use the new scheme.

The new naming scheme is slightly more compact, and copes better with services
that have things like underscores and plus characters (e.g. powerd++) in their
names.

As an ancillary to this, system-control now has an "escape" subcommand which can
be (and indeed is) used in scripts to perform the escaping transformations.

More packages
-------------

There are now four more -shims packages, for commands whose names conflict with
commands from other packages: nosh-kbd-shims, nosh-bsd-shims, nosh-core-shims,
and nosh-execline-shims.

nosh-kbd-shims, for example, contains a chvt shim that is an alias for the (also
new) console-multiplexor-control command; with it, and suitable privileges to
access the virtual terminal's input queue, one can switch between multiplexed
user-space virtual terminals in much the same way as the old chvt command does
with kernel virtual terminals.

The Z Shell command-line completion for the various commands in the toolset
(system-control, svcadm, shutdown, svstat, and so forth), which has been
available to the people building from source for a while, is now also available
as a binary package.

Configuration import
--------------------

ldconfig on TrueOS is now properly handled. In particular, the external
configuration import subsystem now correctly pulls in and converts all of the
ldconfig directories. (TrueOS has a lot more things that require ldconfig
support than stock FreeBSD does.)

The configuration import subsystem also now handles instances of Percona server,
alongside MySQL and MariaDB. Moreover, these are now handled by the same set of
service bundles, which always produce service bundles named mysql@*. MySQL
version 5.7 or later is now assumed.

The configuration import subsystem now automatically generates OpenVPN service
bundles based upon the current OpenVPN configuration.

=======================
==== CAVE: OpenVPN ====
=======================

The upgrade process attempts to remove the old hardwired ***@server and
***@client service bundles. However, you might encounter remnants of these
service bundles lying around in /var/sv that you will find that you need to
clean up by hand.

GOPHER
------

To accompany the new gopherd server in djbwares 5, there is a gopher6d service
bundle that runs it, serving up the same static files area as http6d, https6d,
and ftp4d do.

The FreeBSD, OpenBSD, and Debian package repositories can now be browsed with
GOPHER. This is gopherd in action. On the server side, generating the
index.gopher files is a fairly humdrum exercise in the use of redo (to
regenerate the indexes only when the directory contents change) and printf (to
construct the GOPHER format menus).

UCSPI-UNIX
----------

Two new UCSPI tools have been added to enable UCSPI-UNIX servers to listen on
and accept connections on AF_UNIX sequential packet sockets. udevd is one such
server, and it is now handed its listening socket at startup rather than
expected to open its own.
Jonathan de Boyne Pollard
2017-07-05 20:41:27 UTC
Permalink
The nosh package is now up to version 1.34 .

* http://jdebp.eu./Softwares/nosh/
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

Once again, there are a few more service bundles. The most interesting ones in
this version are perhaps the finish-update and finish-install targets, designed
to be invoked the first bootstrap after an update or install has been done, and
the users target, which is used to auto-start per-user subsystems at bootstrap.
Several NFS service bundles are now common across operating systems. And the
OpenVPN service bundles are now split into separate client and server services.

Several minor bugs have been fixed here and there: a duplicated newline in
line-banner that was throwing off publicfile FTP service; a problem with
recordio on FreeBSD/TrueOS; and a problem with attempts to use slashes in
environment variables in service bundle environment directories.

The user-space virtual terminal emulator now implements the Xterm extensions to
DECSCUSR, and the framebuffer realizer can display the resultant cursor shapes.
This can be made use of by programs such as Neovim.

There are now separate service bundles and nosh-run- packages for running eudev
and systemd-udev, because the two are now significantly divergent.

The various utilities for changing the process environment no longer use the GNU
C library/BSD C library functions for doing so, and so no longer suffer from the
concomitant memory leaks that their manual pages used to warn about.

The convert-systemd-units tool has been slightly enhanced, for the benefit of a
fix that has been made to the per-user gpg-agent service.

The external configuration imports system has been extended. It now deals with
importing the hostname configuration value, taking that responsibility away from
and simplifying the set-dynamic-hostname utility. It now imports various Debian
and other kernel virtual terminal settings, from /etc/kbd/config,
/etc/default/console-setup, and /etc/vconsole.conf . And network configuration
import now can set up services for both dhcpcd and dhclient.
/etc/system-control/convert/rc.conf now contains more settings on Linux
operating systems as a result, including dhclient_program.

Continue reading on narkive:
Loading...