Portfolio-Management

New Project: Portfolio-Management

The Portfolio Manager, when present, would be the entity deciding on:

  1. Base and Quote Asset balances per exchange.
  2. Initial Target Size.
  3. Size Increases.
  4. Size Reductions.

It should have access to actual balances for each asset at exchanges.

At the UI

Portfolio System

This is one of the hierarchies needed:

  • Portfolio System
    • Portfolio
      • [array of] Portfolio Asset > references > Exchange Account Asset
        • Asset Balance
    • Trading Systems
      • [array of] ]Managed Trading System > references > Trading System
        • Managed Base Asset > references > Portfolio Asset
        • Managed Quoted Asset > references > Portfolio Asset
        • Management Events
          • Initial Size Event
            • Formula
          • Size Increase Event
            • Formula
            • [array of] Situation
              • [array of] Condition
                • Javascript Code
          • Size Decrease Event
            • Formula
            • [array of] Situation
              • [array of] Condition
                • Javascript Code

In words:

A Portfolio Manager has Portfolio and Trading Systems to manage. The Portfolio is essentially a collection of Portfolio Assets, which by referencing Exchange Account Asset nodes they get well defined, since in turn those nodes reference Exchange Assets and their parents know which keys to use to access them at exchanges.

On the other side, the Trading Systems managed is just a list of Managed Trading Systems, each one declaring its own Managed Base and Quoted Assets.

Finally we have the 3 events that need the Portfolio Manager to take some action. Initial Size, Size Increase and Size Decrease.

Initial Size happens when a managed Strategy wants to take position, meaning that it does not depend on the Portfolio Manager, for that reason it does not have a collection of Situations / Conditions like the 2 other events, the Size Increase and Size Decrease.

All events have a Formula which allows users to freely define the Size based on any of the information available, including the Asset Balances of each of the Portfolio Assets at exchanges.

Portfolio Mine

This is the hierarchy equivalent to the already existing Data Mine, Trading Mine and Learning Mine.

  • Portfolio Mine
    • [Array Of] Portfolio Bot
      • Same children as other types of bots.
    • [Array Of] Plotter

Portfolio Engine

This is the hierarchy equivalent to the already existing Trading Engine. It is the data structure that the Portfolio Bot will use in-memory for its own processing.

  • Portfolio Engine
    • … pending definitions …

Here we need to design the data structure that we want to keep in memory at the Portfolio Bot. This data structure needs to have all the information that the bot uses for doing its job, plus statistics that later users will be able to plot at the charts, see at panels or with the visual debugging capabilities of the system.

At the moment I imagine that the Portfolio Bot will have access to the data of all Trading Bots it is managing via reading their output files like indicators. That would be the raw data which the Portfolio Bot makes decisions on. A second set of raw data is the information the Portfolio Bot can get from exchanges. The most obvious ones are asset balances at each exchange. Based on those 2 sets of raw information, the Portfolio Bot should populate its own Portfolio Engine in-memory data structure.

Then it would evaluate if any of the events that it manages (Initial Size, Size Increase or Size Decrease) needs to be processed. The Formulas defined at these events, will also have access to the raw data coming in the form of indicators and trading data from Trading Bots. At the same time, they will also have access to the Portfolio Engine in-memory data structure and use all of the previous in calculating the Sizes, evaluation Situations and Conditions.

Finally the Portfolio Bot will have its own output. Part of it will be read by Trading Bots in a similar way as Indicators are, and from there Managed Trading Bots will know of the Initial Size, Size Increases and Size Reductions. Today Trading Bots calculates their own Initial Size and there is no way to trigger Size Increases or Reductions. Those features need to be added.

Data Dependencies

Today a Trading Bot has an explicit declaration of all possible Data Dependencies, which are nodes referencing each Dataset of each Indicator of every Data Mine existing.

An Indicator Bot works exactly in the same way, it has an explicit declaration of which Datasets it depends on. The difference with Trading Bot is that an Indicator bot only declares what it actually needs. The Trading Bot declares them all, since it does not know, at design-time which indicators users will be finally using at their strategies.

Because Trading Bots have them all, we needed the concept of Dependency Filters. What is that? We analyze the syntax of all Formulas and Conditions at a Trading System and from there extract a list of the Indicators actually used. So, at a Trading Bot, instead of loading all Datasets declared at design-time, we avoid loading all the ones that have not been mentioned in Conditions and Formulas (we apply the dependency filters).

A Portfolio Bot also needs Indicators, since they might be used at its own Conditions and Formulas. But it will also need Trading Bots outputs, which means that we need to declare at design-time, data dependencies pointing to the Datasets outputted by Trading Bots that are needed for the Portfolio Bot to make its decisions.

Here I can see on upgrade we will need to do to the current infrastructure:

Today these data dependencies assume that there is for instance, only one Candles Dataset, or only one Bollinger Bands dataset. What I mean is that there are not two different instances of the Candles Indicator running at the same time producing Candles. This is not the case of Trading Bots.

The Portfolio Bot needs to be able to manage any number of Trading Bot instances, but the current declaration of data dependencies just point to the Trading Bot’s generic Dataset definition. That means that we will need to upgrade this to handle the situation that now we will need many Datasets from a single declaration of Datadependency if they are pointing to a Trading Mine dataset.

Tasks

The last part we need to nail is the nodes needed to launch Portfolio Bots tasks.

The problem to solve here is how do we relate the Portfolio Task to the Trading Tasks. I will have to think about this to get to a clean solution.

Open Questions

  1. Will we have backtesting of Portfolio Management? No, reserve for version 2.

Conclusion

The previous setup is not simple to pull out, but we have the blueprint of the Trading Bots to follow. If we make it in this way, this will be quite powerful since a single Portfolio Manager, could be managing:

  • Unlimited number of assets.
  • At an unlimited number of exchanges.
  • Being traded by an unlimited number of Trading Bots.
  • Running in an unlimited number of different machines.

It will also leverage the knowledge of SA users of concepts like the Trading System, Trading Engine, Trading Mine, which now turns into Portfolio System, Engine and Mine, allowing the user to have charts related to Portfolio Management and also Visual Debugging of anything the Portfolio Bot does.

Next Steps

ok, how shall we continue?

I see 2 avenues:

  1. I lay down all the infrastructure myself. I don’t know how you can use your time in the meantime. I believe this is a beautiful project and would like to do it myself, but I recognize that it makes me diverge from my priorities, that is the p2p network and social apps.

  2. Tell you what I would do and step by step, you do it. While you do it I can think of the parts that are missing. I could invest some daily time on this to avoid roadblocks.

Milestones / Roadmap:

  1. Forking Algorithmic-Trading Project

The first milestone is to add a new project Portfolio-Management, which means forking the current project Algorithmic-Trading and refactoring Trading for Portfolio everywhere.

Milestone is reached once that fully works as a Trading Bot.

This means allowing the Portfolio Bot to have as data dependencies the output of the Trading Bots. We haven’t done this before, so there is some innovation to do there.

Step by Step:

  1. Copy the Algorithmic-Trading Folder into Projects and rename it Portfolio-Management
  2. Open a VS Code on that folder so that all refactoring is constrained there.
  3. Refactor considering upper and lower case: Trading → Portfolio ; trading → portfolio
  4. Inside the file Project Schema, also copy the Algorithmic Trading section into Portfolio, and refactor the new section only.
  5. Be careful because VS Code search does not include file names, which also needs to be refactored.
  6. Take care of the code that mentions Trading outside the Algorithmic Trading folder:

So here we will be standing at the point where we have done the refactoring, but there is a problem: I expect many things related to algorithmic trading to not be inside this project, since the extraction of the Algorithmic Trading project from inside the Foundations project was not perfect or lets say, not 100% complete.

To find out what is missing, I would search the whole Superalgos Codebase for trading, and see what I can see outside the Algorithmic-Trading folder. Depending on what I see, are the following actions to take.

Just removed the folder Algorithmic-Trading from my local and I have 25 files to check why they have the trading word inside:

I would go one by one and see if there is something relevant. I am looking for things like places where Trading Tasks are started or something like this. It might happen that nothing there is relevant and there is nothing to change at all.

If I find something relevant I will probably need to do or add something to consider the new Portfolio Bots.

  1. Once that job is over, the next thing to do is to run the Platform and see if I can add a Portfolio Task, and run it. Most likely Many things will not work, so we will have to fix them to get to the point where we can run a Portfolio Task.

  2. The things I expect not to work are configs at APP SCHEMA files. Once sorted out we would be able to run the task.

  3. Running the task is cool, next we need to run is a Session. The refactoring inside the would have the bot expecting a Portfolio System instead of a Trading System, a Portfolio Mine, instead of a Trading Mine, and a Portfolio Engine instead of a Trading Engine.

  4. Here I see some problems if the refactoring did not include file names.

  5. Another thing to consider is the list of Plugins defined for each project at the Project Schema file.

  6. If you add a new Portfolio Mine, a new Portfolio Engine and new Portfolio System, and reference those from the Session, then at some point it should work.

  7. After solving all the previous problems, we would have a Portfolio bot running, with the business logic of a Trading Bot. That leaves us ready to remove that logic and code the logic of the Portfolio Bot.

  8. Resolving the Data Dependencies of the Trading Bot

This means allowing Trading Bots to have access to the output of the Portfolio Bot.

  1. Coding the logic of the Portfolio Bot

This means coding the business logic we already talked about.

  1. Upgrading Trading Bots

To be able to work with Portfolio Bots when needed. Add features to ask for size, request size increase, request size decrease, execute a size increase / decrease when needed to.

3 Likes

Once we have the Portfolio Bot running with the logic of a trading bot the next important steps are:

A.

We need to add to the current Task node a new child type: Managed Tasks (#2 at image / project Foundations). Each managed task should have an array of child nodes Task Reference (new). (#3 at image / project Foundations)

This will allow us to run all the tasks that are going to be managed automatically every time users press Run at the Task node (and to stop them all when they press stop).

A similar trick needs to be implemented at the Management Session node, when users press run, all the sessions at of the managed tasks should also be run, and stopped when users press stop.

After finishing implementing this, we will be able to run the Portfolio Task and all trading bots task, no matter at which machine they are, they will also run.

The beauty of this approach is that Trading Bots will still be able to be ran independently, for backdating or live trading. They will only be managed when they are run from the Portfolio Task.

B.

These new nodes information needs also to be available at the Portfolio Task Nodejs Process, in order to communicate the Portfolio Bot with the Trading Bots.

To achieve that, you will need to modify this file to add the nodes that needs to be sent to the Task Server:

At the Task Server we will be able to know for instance if the managed Tasks are alive or not, since we will be able to hear to their heartbeats in a similar way that the UI currently hear to those heartbeats in order to show at the screen the circular progrss bar that shows today when a Task is running.

I would not go right now to code the monitoring of the managed tasks themselves but I am pretty sure this will be needed down the road.

Once we get this changes at TasksFunctions.js implemented we would have at the Task Server the concept of managed tasks, in the form of available information of all the Tasks being managed. With this I mean we could have the nodes structures of each of the managed tasks and know which are their default exchanges, session parameters, etc. Some of that stuff will be needed down the road.

C. Sessions

Once we have the task layer figured out. the next level are sessions. For this use case we will need a new time of Session.

I will expand this line of thoughts in a next post.

Portfolio System:

1.) For the Portfolio System hierarchy are you envisioning something like this below? I was unsure if the individual portfolio_assets needed to each be referencing their own Exchange_Assets->Asset or if the portfolio itself would just reference the Exchange_Assets node and ‘inherit’ the assets found there?

Milestones:

Status:

  • Forking Algorithmic-Trading
    • Steps 1 - 6 Complete
    • Step 7 ) All added to Workspace.
      QUESTION: You say “run it” do you mean Portfolio Tasks should exist at ‘LAN Network Node’ between Data_Tasks and Learning_Tasks or should it exist as child of Data_Tasks or somewhere else?
    • Step 8) Pending, should be working.
    • Step 9) Pending step 7 completion
    • Step 10) Complete
    • Step 11) Unsure what this referring to. In your next post you mention the Task server and I have added the references there already during the refactoring steps and I have familiarity with this system, believe it will work or minor modification will make it so.

Post #2:

Step A will be easy and I believe Step B is already complete but what I am unsure of is what you mean by this sentence:

  • I would not go right now to code the monitoring of the managed tasks themselves but I am pretty sure this will be needed down the road.

Step C) You would have to elaborate more on this after all of the above is complete (or whenever you feel like it of course.)

1 Like

It is better that each individual Proftolio Asset references one Asset. Why?

The node Exchange Assets children represents all the assets users would like to use at that workspace from that Exchange Not necessary all the assets an specific Portfolio Manager would manage.

On the other hand, a Portfolio Manager could be managing at the same time assets from multiple exchanges.

1 Like

Yes I mean that Portfolio Tasks should exist between Lata Tasks and Learning Tasks. This whole structure like in this picture should exist in order for the Portfolio Task to be able to run (with the logic of a trading bot).

Once you can run the Portfolio Bot Task and Session and all is working we might revisit this sequence of nodes and potentially adapt them to what the Portfolio Bot actually needs, but in the meantime, keeping the structure is going to help us reach the milestone of the bot running which will guarantee that we have a floor where to stand to build the rest.

For the Portfolio System, the node structure I see it is still the one of the original post:

I think you can implement this via creating the needed nodes APP SCHEMA files and we can check them out at the APP once that is setup.

1 Like

What I meant is that do not forget these definitions at the Project Schema File, so that the Portfolio System, Mine and Engine are considered plugins types for this project.

Ok let me explain this.

The concept of Managed Tasks we are introducing here does not belong to the Portfolio-Management project itself, but to the Distribute-Tasks project. This feature of a Task having managed task is an infrastructure-level work you will be doing, which the Portfolio-Management project will be the first use case benefiting from this, but potentially not the only one in the future.

Here is the still not extracted from Foundations Distributed-Tasks project introduced at my blog post describing all proejcts.

So in this context I was saying that at the Task Server, a Task that do have managed Task, potentially could monitor the managed ones are running and if not maybe act on consequence, doing something, like alerting or running them again, or who knows. All this stuff we are not going to do for now, because it diverges from our main focus of putting Portfolio-Management to work.

1 Like

There was a typo here, I meant a new Type of Session, not a new Time of Session, but yes, indeed, once we get close to this point I will expand it.

Yes this has been done per step 4
image

That makes sense. You scared me with the word time.

Hope the idea of exchange assets could be classified as manually traded and bot available funds. Helps users migrate over to bot only ( or else a clean account would be necessary )

1 Like

Don’t know if its ok but attaching a Portfolio config (fyi), asset wise distribution.

And Portfolio Rules

Good idea, each of these exchange assets should have at their config a property maxAvailableFunds or something like this.

2 Likes

Node actions/functions extremly powerful extension capabilities. We could as-is read the exchange assets (thru ccxt). (Over even adjust margin, pay/transfer assets…well atleast later on during the product lifetime :wink: )

We are done with #13 of part 1. I will quickly go to add Part A’s nodes of step 2 then
The question here is should I do step 14 of part 1 prior to re-writing the logic? My intuition is ‘no’ because otherwise could end up doing same/similar work twice.
So now the question becomes which should be next:

  • Opt1: Begin figuring to code logic of portfolio bot
  • Opt2: Deal with the Part A, while portfolio bot is working as trading bot clone, to implement the task references and functionality to run all managed tasks.
  • Opt3: Go ahead and revise refactoring to include backtesting portfolio tasks. Then there is also the question of I assume a paper portfolio session should also be included when the backtesting session is added?

Also, will like to save the whole distributed tasks part for later/last.



image

While you are here could you briefly explain what is the thought process / motive behind splitting some of the actionSwitch 's out of foundations into visual-scripting. I think i’m starting to understand which type is moved to visual-scripting but am unsure so to be clear what types go in visual scripting vs foundations?

I believe these are the sorts of tasks should be managed by, turned on|off by, the PM’s Managed Tasks. Is that similar to how you were thinking?

image

Good question. I don’t see a very clear boundary. All I know that Foundations is where all started and still have a mix of different dimensions of the project. After we extract all the dimensions we can recognize, there might be something left in there or probably nothing.

We can define Visual Scripting to the features of the system that allow us to have the concept of nodes and their relationships:

Parent Nodes
Reference Nodes

and all actions that we can do with them, like attach them, or reference each other.

I guess that having configs at individual nodes are also one aspect of Visual Scripting.

There are quite a few utilitiy functions related to operations with nodes, like finding missing children for instance that are clearly part of Visual Scripting.

Respect to Action Scrip Switch let’s first define an Action Script:

An Action Script would be a script that can be accessed by a Menu Item of some node, which intention is mainly manipulate the nodes structures at the workspace automating tasks that users otherwise would need to do manually.

For example Install a Market. This can be done manually connecting hundreds of stuff, or someone could automate that into an Action Script.

If the task being automated is related to a project: for example Install Product is a task recentrly automated and allow people building indicators not worry about adding the references to a new product to the Trading Data Mine. That action script is automating something clearly related to Data-Mining Project, so anyone should expect the code to be accesible from the Action Switch of the Data-Mining Project, and the module with the actual code of that action script to be also at the Data Mining project.

Today there are hundreds of action script still at Foundations which belong to other projects, just because nobody have ported them yet.

1 Like

Yes, this looks good to me.