How to create your own private messages service with macOS Server!

How to set up iMessage on your Mac (Image credit: Joseph Keller / iMore)

Instant messaging is a must in today's business sharing and collaboration software toolset. There are instant messaging services that are free to use (but may need access to your data) or there are paid services that can charge per user.

If you're in need of an instant messaging service that avoids giving up your private information and only costs a nominal fee regardless of the number of users you need, then the messages service on macOS server may fit your bill.

We'll be showing you how to setup and create a private messaging service on macOS for your locally networked connected Macs, iOS devices, and PCs connected to your macOS Server VPN service. Although we can set up this service to allow access from anywhere on the internet without a VPN, we feel the purpose of offering a highly secure and private messaging service outweighs the convenience of being easily accessible.

Install macOS Server on one of your Macs

If you haven't already done so, you'll need to install macOS Server on one of your network Macs. You can follow our detailed guide if you need a little more help, but here is the express version.

Launch the App Store application.
Search for macOS Server in the search bar at the top right of the app.
Click Buy next to macOS Server or if already purchased, you'll be notified that you can download it for free.
Click Install.
Agree to the User Agreement.

Add users to your macOS Server

In order for your organization to start collaborating, you'll need to add the users whom you want to have access to the service.

Start the Server App.
Select Users from under the Accounts listing.
Click the Add button (it looks like a plus symbol) to add a new user.
Type in the Full Name of the user you want added.
Enter a Password for the user.
Verify the password for the user.
Click Create.

Repeat for as many users as required.

Configure the messages service

We'll now need to configure the messages service to only accept connections from your local network IP addresses.

Select Messages from the services listings in the Server App.
Click Edit Permissions.
Select private networks from the dropdown list under When connecting from.
Click OK.

Optionally check Archive all messages if you wish to save all of your organization's messages from every user to disk.Note: If you wish to secure this data, make certain to select an encrypted location to save this file to.
Toggle the ON/OFF switch to ON.

If you have an Airport Base Station, you will be prompted to allow for macOS Server to automatically open ports through your router so that external clients can connect to your messages server. Do not allow this. Although it's possible to set this up as a regular internet service to connect from anywhere, the purpose of this tutorial to keep it as secure and private as possible and that sort of setup is beyond the scope of this article.

Connect your Mac clients

For macOS, Apple has already built in functionality for the xmpp protocol, the software that underlies the messaging system used by macOS Server. This means that you needn't download any third party messaging clients and you can work seamlessly with the messages app.

Start your Messages app.
Click Messages in the App menu in the upper left corner of the screen.
Select Add Account from the drop-down menu.
Select Other Messages Account.
Click Continue.

Select Jabber from the Account Type dropdown list.
Enter the User Name you created on your macOS Server in the form of UserName@mac-mini-host-name.local.Note: It's very important to use the Host Name associated with your macOS Server otherwise you'll need to edit configuration files located in /Library/Server/Messages/Config/jabberd.
Enter the Password associated with the user name you entered.
Click Sign In.
Click Continue when prompted to accept the server certificate.

Set your Jabber status to available to log in.
Select a user on the same Local Account as the private server you connect to to start messaging.

Connect your iOS device

You can do a search for xmpp from the app store to see a list of xmpp clients available to download but we found that Monal is about as easy to set up as can be and works well with our macOS Server. Monal also runs on Android and has a native macOS client as well.

Open the App Store on your iOS device.
Search for Monal.
Tap Download.
Tap Install.
Tap Open.
Skip the Tutorial.
Add a New Account.
Under Jabber ID enter your UserName@macOS-server-host-name.local.Note: Make certain to enter a User you created on the messages server and also make certain you use the host name you used when setting up your macOS Server.
Set Self-Signed Certificate to on.

Connect your PCs

We won't go too deeply as to how to connect your Windows and linux PC's but xmpp.org has a nice list of different clients you can try to connect to your macOS messages server.

Connecting when on the local VPN

For you remote users, you can still take part in the private messaging service through the VPN (virtual private network) if you have one set up. You'll need to add your macOS Server's internal network IP address to your personal computer's Host file. Here's how you do it on macOS.

In finder or terminal navigate to /etc.
Edit hosts using your favourite text editor using an administrative account.
Append the IP Address and Host Name of your macOS Server. This will be in the form of "xxx.xxx.xxx.xxx Hostname.local". For example, mine looks like "192.168.15.154 anthonys-mac-mini.local".